Program, information processing method, and information processing device

Abstract

To provide a program etc. capable of accurately extracting a card area from a photographed image of a card.SOLUTION: A computer obtains a photographed image in which a card is captured. Then, the computer performs, to a photographed area of the card cut out from the photographed image in which the card is captured, on which personal information is printed, template matching using a template image in which an area of the personal information is masked, for extracting a photographed area of the card within the obtained photographed image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present application relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] In recent years, various procedures have been conducted online, and ID (identification) cards, which serve as identification or personal verification documents required for the procedures, are photographed with a camera such as a smartphone and then submitted (transmitted) via a network. A device that receives the photographed image of the ID card performs authentication of the ID card based on the photographed image. To enable submission of an ID card using a photographed image, it is necessary to appropriately extract the captured area of ​​the ID card from the photographed image. For example, Patent Document 1 discloses a device that extracts a closed area of ​​a continuous color from a photographed image and determines a card area to be cut out from the photographed image based on the difference between the extracted closed area and a template. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-140352 Summary of the Invention [Problem to be solved by the invention]

[0004] When photographing an ID card at home, for example, the lighting conditions or background pattern at the time of photographing can make it difficult to accurately extract the card area from the photographed image. Furthermore, some ID cards have holograms on their surfaces as a security measure. With such ID cards, natural or illuminating light is reflected by the hologram during photographing, making it difficult to capture an image from which the card area can be accurately extracted. With conventional technologies such as those disclosed in Patent Document 1, it is difficult to accurately extract the card area from an image captured in such a state where light is reflected by the hologram. Furthermore, some ID cards have patterns printed on their surfaces. With photographed images of such ID cards, it is difficult to accurately extract closed areas with a continuous color, and the technology disclosed in Patent Document 1 cannot adequately address this issue.

[0005] The present disclosure has been made in consideration of the above circumstances, and its purpose is to provide a program or the like that is capable of extracting a card area from a photographed image of a card with high accuracy. [Means for solving the problem]

[0006] A program according to one aspect of the present invention causes a computer to acquire an image of a card, and extract the photographed area of ​​the card from the acquired image by template matching a template image in which the area of ​​the personal information is masked against the photographed area of ​​the card cut out from the photographed image of the card having personal information printed on its face. [Effects of the Invention]

[0007] According to one aspect of the present invention, a card area can be extracted with high accuracy from a photographed image of a card. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of an information processing system. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of a server and a user terminal. [Figure 3] 10 is a flowchart illustrating an example of a registration process procedure for a template image. [Figure 4] FIG. 10 is a schematic diagram showing an example of a screen. [Figure 5] FIG. 10 is an explanatory diagram of a registration process of a template image. [Figure 6] 10 is a flowchart illustrating an example of a procedure for extracting a face area of ​​a card. [Figure 7] 10 is a flowchart illustrating an example of a procedure for extracting a face area of ​​a card. [Figure 8] FIG. 10 is a schematic diagram showing an example of a screen. [Figure 9] FIG. 10 is an explanatory diagram of a process for extracting a face area of ​​a card. [Figure 10] FIG. 10 is an explanatory diagram of a process for extracting a face area of ​​a card. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a program, an information processing method, and an information processing device according to the present disclosure will be described in detail with reference to the drawings illustrating embodiments thereof.

[0010] 1 is an explanatory diagram showing an example of the configuration of an information processing system. In this embodiment, an information processing system is described in which a user photographs an ID card using the camera of their user terminal 20, transmits the obtained photographed image to a server 10, and extracts a photographed area of ​​the ID card from the photographed image received by the server 10. The ID card in this embodiment includes cards and documents that can be used as identification or personal verification documents, such as a driver's license, personal identification card (My Number card), residence card, health insurance card, passport, etc.

[0011] The information processing system of this embodiment includes a server 10 and a user terminal 20, and the server 10 and the user terminal 20 are communicatively connected via a network N. The network N may be the Internet or a local area network (LAN) constructed within a facility where the information processing system is installed. The server 10 is an information processing device capable of various information processing and transmitting and receiving information, such as a server computer or a personal computer. The server 10 acquires a photographed image taken by the user terminal 20 using a camera 26 (see FIG. 2 ) and performs a process of identifying and cutting out a photographed area of ​​an ID card appearing in the acquired photographed image. The user terminal 20 is an information processing device capable of various information processing and transmitting and receiving information, such as a smartphone, a tablet terminal, a personal computer, etc. The user terminal 20 is equipped with a camera 26 and performs a process of photographing an ID card using the camera 26 and transmitting the acquired photographed image to the server 10.

[0012] 2 is a block diagram showing an example configuration of the server 10 and the user terminal 20. The user terminal 20 has a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, a display unit 25, a camera 26, etc., and these units are connected via a bus. The control unit 21 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), or a GPU (Graphics Processing Unit). The control unit 21 executes a program 22P stored in the storage unit 22 as needed to perform information processing and control processing that the user terminal 20 should perform.

[0013] The storage unit 22 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), etc. The storage unit 22 stores a program 22P (program product) executed by the control unit 21 and various data. The storage unit 22 also temporarily stores data and the like that is generated when the control unit 21 executes the program 22P. The program 22P and various data may be written to the storage unit 22 during the manufacturing stage of the user terminal 20, or may be downloaded by the control unit 21 from another device via the communication unit 23 and stored in the storage unit 22. A part of the storage unit 22 may be another storage device connected to the user terminal 20.

[0014] The communication unit 23 is a communication module for performing processes related to wired or wireless communication, and transmits and receives information to other devices via the network N. The input unit 24 accepts operation input by the user and sends a control signal corresponding to the operation content to the control unit 21. The display unit 25 is a liquid crystal display, an organic EL display, or the like, and displays various information in accordance with instructions from the control unit 21. A part of the input unit 24 and the display unit 25 may be a touch panel configured as an integrated unit.

[0015] Camera 26 takes a photograph in accordance with instructions from control unit 21, and acquires, for example, one piece of image data (still image). The image data (captured image) acquired by camera 26 is stored in storage unit 22. Camera 26 may be configured to be built into user terminal 20, or may be configured to be externally attached to user terminal 20. In this case, user terminal 20 includes a connection unit to which an external camera can be connected or a wireless communication unit capable of wireless communication with the external camera, and acquires the image captured by the external camera via the connection unit or wireless communication unit.

[0016] The server 10 includes a control unit 11, a memory unit 12, a communication unit 13, an input unit 14, a display unit 15, etc., and these units are connected via a bus. The above-mentioned units 11 to 15 of the server 10 have the same configuration as the units 21 to 25 of the user terminal 20, so a description of their configuration will be omitted. Note that the memory unit 12 of the server 10 stores, in addition to various programs 12P (program products) executed by the control unit 11, a card face cutting program 12a and a template image 12b for realizing the process of cutting out the captured area of ​​an ID card from a captured image of the ID card. Note that the template image 12b may be incorporated into the card face cutting program 12a.

[0017] In this embodiment, server 10 may be a multi-computer consisting of multiple computers, a virtual machine virtually constructed by software, or a cloud server. Server 10 does not necessarily include input unit 14 and display unit 15. It may be configured to accept operations through a connected computer or to output information to be displayed to an external display device. Server 10 may also include a reading unit that reads non-transitory computer-readable portable storage medium 10a, and may read program 12P from portable storage medium 10a using the reading unit and store it in storage unit 12. Program 12P and card face cutting program 12a may be executed on a single computer or on multiple computers interconnected via network N.

[0018] The following describes the processing performed by each device in the information processing system of this embodiment. First, the processing performed by the server 10 to register a template image used in a card face extraction process that extracts an ID card photographed area from a photographed image of the ID card will be described. Fig. 3 is a flowchart showing an example of the template image registration processing procedure, Fig. 4 is a schematic diagram showing an example screen, and Fig. 5 is an explanatory diagram of the template image registration processing.

[0019] In order for the server 10 to execute the card face cutting program 12a to perform the card face cutting process, a template image of the ID card to be cut out must be registered. In the following process, a user of the server 10 registers a template image of the ID card to be set as the cutting target to the server 10. For example, the user instructs the server 10 to execute the template image registration process via the input unit 14, and inputs a photographed image of the ID card to be registered in the template image to the server 10. The photographed image of the ID card to be registered in the template image is photographed or read by any camera or scanner, and the obtained photographed image is input to the server 10 via, for example, the communication unit 13 or the input unit 14. Therefore, the control unit 11 of the server 10 acquires a photographed image of the ID card to be used as the template image via, for example, the communication unit 13 or the input unit 14 (S11). It is desirable that the photographed image at this time be an image of the ID card photographed or read from the front.

[0020] When instructed to execute the template image registration process, the control unit 11 displays a template image registration screen on the display unit 15 and displays the acquired captured image of the ID card on the registration screen (S12). FIG. 4A shows an example of a registration screen displaying a captured image of an ID card. The captured image of the ID card includes the captured area of ​​the ID card (hereinafter also referred to as the face image) and the surrounding background area. The ID card shown in FIG. 4A has personal information printed on its face, such as the card number, name, date of birth, place of residence, date of issue, and facial photograph, but the personal information printed on the ID card is not limited to these. The screen shown in FIG. 4A has a "Set Face Image" button for instructing setting of the ID card's captured area (the area of ​​the face image) for the displayed captured image. On the screen shown in FIG. 4A, the user specifies the area of ​​the face image by a predetermined operation (e.g., dragging the mouse) via the input unit 14 and operates the "Set Face Image" button.

[0021] When the control unit 11 receives an operation on the "Set Card Image" button on the registration screen, it accepts the designation of the area of ​​the card image via the registration screen (S13). In the example shown in FIG. 4A, a rectangular area indicated by a thick solid line is designated as the area of ​​the card image. The control unit 11 extracts the designated area of ​​the card image from the photographed image of the ID card and generates the card image (S14). At this time, the control unit 11 updates the image displayed on the registration screen to the generated card image. FIG. 4B shows an example of the registration screen displaying the card image. The screen shown in FIG. 4B has a "Set Personal Information Area" button for instructing the user to set the area of ​​personal information printed on the face of the ID card in the displayed card image. On the screen shown in FIG. 4B, the user performs a predetermined operation via the input unit 14 (e.g., dragging the mouse) to designate the area of ​​personal information printed on the face of the ID card in the card image, and then operates the "Set Personal Information Area" button.

[0022] When the control unit 11 receives an operation on the "Set Personal Information Area" button on the registration screen, it receives the designation of the printing area of ​​the personal information in the card face image via the registration screen (S15). In the example shown in FIG. 4C, the printing areas of the card number, name, date of birth, place of residence, date of issue, and facial photograph in the card face image are each designated as a rectangular area. The control unit 11 acquires the coordinate values ​​of the printing area of ​​each personal information based on the area of ​​each personal information designated for the card face image (S16). For example, as shown in FIG. 5A, the control unit 11 acquires coordinate values ​​(x, y) indicating the bottom right position of the card face image and the top left and bottom right positions of the printing area of ​​each personal information, with the top left position of the top left of the card face image as the origin (0, 0), the X axis extending to the right from the origin, and the Y axis extending downward from the origin. In the example of Figure 5A, the coordinate values ​​(X, Y) of the bottom right position of the card face image, the coordinate values ​​(x1, y1) of the top left position and the coordinate values ​​(x2, y2) of the bottom right position as the coordinate values ​​of the name printing area, and the coordinate values ​​(x3, y3) of the top left position and the coordinate values ​​(x4, y4) of the bottom right position as the coordinate values ​​of the card number printing area are obtained.

[0023] Based on the coordinate values ​​of the acquired card face image and the coordinate values ​​of the printing area of ​​each personal information, the control unit 11 generates a mask image for the card face image by masking the printing area of ​​each personal information (S17). Fig. 5B shows an example of the mask image, and when each pixel value in the image is expressed in 256 gradations, for example, a mask image is generated in which the pixel value of the white area in the mask image (area other than the mask area) is set to 255 and the pixel value of the mask area is set to 0.

[0024] Next, the control unit 11 performs downsampling on the face image generated in step S14 (S18). For example, the control unit 11 performs downsampling using a nearest neighbor algorithm, bilinear algorithm, bicubic algorithm, or other algorithm to reduce the size of the face image and reduce the amount of data. The control unit 11 then performs masking on the face image after downsampling based on the mask image generated in step S17 to generate a template image in which the personal information area in the face image is masked (S19). Figure 5C shows an example of a template image, and a template image is generated in which the personal information area in the face image, such as that shown in Figure 4B, is masked. In Figure 5C, the hatched areas indicate the masked areas. Note that the face image has been reduced in size by downsampling, and using a template image generated from such a face image can shorten the processing time for the matching process. In the template image shown in Figure 5C, the background of the face image (the background of each personal information) is shown in white, but the background area contains the pattern of each ID card.

[0025] The control unit 11 stores the generated template image in the storage unit 12 (S20), and the series of processes ends. The above-described process allows registration of template images of ID cards to be cut out when the server 10 executes the card face cutting process. The user registers template images by performing the above-described process for each ID card, such as a driver's license, personal identification number card, residence card, health insurance card, and passport. This allows each registered ID card to be used as a cutout target in the card face cutting process by the server 10. The processes of steps S18 to S19 in the above-described process may be performed before the template image is used. In this case, the control unit 11 stores the card face image generated in step S14 and the mask image generated in step S17 in the storage unit 12. When using the template image, the control unit 11 may then read the card face image and the mask image from the storage unit 12, perform downsampling on the card face image, and perform masking based on the mask image on the downsampled card face image to generate the template image.

[0026] Next, we will explain the process in which the server 10 extracts the ID card photographed area (face area) from the image of the ID card photographed by the user terminal 20, using the template image registered by the above-mentioned process. Figures 6 and 7 are a flowchart showing an example of the procedure for extracting the face area, Figure 8 is a schematic diagram showing an example screen, and Figures 9 and 10 are explanatory diagrams of the face area extraction process. In Figure 6, the left side shows the process performed by the user terminal 20, and the right side shows the process performed by the server 10. The feature point matching process shown in Figure 7 is the process of step S40 in Figure 6.

[0027] The following describes the process of extracting the card face area from the captured image by the server 10 when, for example, a user submits a captured image of an ID card when performing various procedures via the network N using the user terminal 20. Note that the server 10 may be a server that performs a process to accept procedures, or may be a server that only performs a process to extract the card face area from the captured image.

[0028] The control unit 21 of the user terminal 20 determines whether an instruction to execute a photographing process has been received through an operation via the input unit 24 (S31). If it determines that an instruction to execute a photographing process has not been received (S31: NO), the control unit 21 waits while performing other processes. If it determines that an instruction to execute a photographing process has been received (S31: YES), the control unit 21 takes a photograph using the camera 26 and acquires a photographed image of the ID card (S32). If a photographed image of the ID card has been acquired, the control unit 21 displays the acquired photographed image on the display unit 25 (S33). FIG. 8A shows an example of a display screen for a photographed image. The screen shown in FIG. 8A displays the photographed image of the ID card and has a “Retake” button for instructing the user to retake the photographed image and an OK button for instructing the user to submit the displayed photographed image. The user of the user terminal 20 checks the displayed photographed image, determines whether the photographed image can be used for submission, and operates either the “Retake” button or the OK button.

[0029] The control unit 21 determines whether or not an operation on the OK button on the screen shown in Fig. 8A has been accepted (S34), and if it determines that the operation has not been accepted (S34: NO), that is, if an operation on the "take again" button has been accepted, the process returns to step S31. Then, the control unit 21 takes a photograph in accordance with the instruction to execute the photographing process accepted via the input unit 24, and displays the acquired photographed image. If it determines that an operation on the OK button has been accepted (S34: YES), the control unit 21 transmits the photographed image of the ID card being displayed to the server 10 (S35).

[0030] The control unit 11 (acquisition unit) of the server 10 receives the captured image of the ID card sent from the user terminal 20 and performs downsampling processing on the received captured image (S36). Here, the control unit 11 performs downsampling processing at the same compression rate as the downsampling processing performed on the card face image in step S18 during the processing shown in Fig. 3. This reduces the amount of data of the captured image to be processed, thereby shortening the processing time.

[0031] Next, the control unit 11 performs noise removal processing on the downsampled captured image by filtering using a smoothing filter such as a Gaussian filter or a bilateral filter (S37). Note that when a bilateral filter is used, the image can be smoothed while retaining edge information in the image.

[0032] Next, the control unit 11 performs keypoint matching (template matching) based on the captured image after noise removal processing and a template image previously generated by the processing shown in Fig. 3 and stored in the storage unit 12, to associate each feature point in the template image with a corresponding feature point in the captured image. Keypoint matching can be achieved using keypoint matching techniques such as SIFT (Scale Invariant Feature Transform), ORB (Oriented FAST and Rotated BRIEF), BRISK (Binary Robust Invariant Scalable Keypoints), KAZE, and AKAZE (Accelerated KAZE). Note that the applicant has confirmed through experiments that keypoint matching using SIFT features has good accuracy and stable processing, so processing using SIFT features will be described below.

[0033] In keypoint matching, the control unit 11 first detects feature points in the captured image after noise removal processing and calculates feature amounts of the detected feature points (S38). For example, the control unit 11 uses OpenCV (Open Source Computer Vision Library) to execute processing according to the SIFT algorithm to detect feature points in the entire region of the captured image and calculate SIFT feature amounts for each feature point. Here, the control unit 11 calculates, for example, 128 feature amounts for each feature point. The control unit 11 also executes processing according to the SIFT algorithm using OpenCV on the template image to detect feature points in the template image and calculate feature amounts for the detected feature points (S39). Here, the control unit 11 detects feature points in the region of the template image other than the mask region and calculates SIFT feature amounts for each feature point. The detection of feature points in the template image and the calculation of SIFT feature amounts only need to be performed once. By storing the detection results and calculation results in the storage unit 12, the processing of step S39 can be omitted from subsequent processing.

[0034] Next, the control unit 11 performs a matching process in which the feature amounts of each feature point calculated for the captured image are compared with the feature amounts of each feature point calculated for the template image, and the feature amounts of each feature point are associated with the feature points in the captured image (S40). In the feature point matching process shown in FIG. 7, the control unit 11 extracts one feature point from the template image (S51). Then, the control unit 11 identifies (matches) a feature point in the captured image whose feature amount is closest (similar) to the extracted feature point of the template image using, for example, FLANN (Fast Library for Approximate Nearest Neighbors). Specifically, the control unit 11 calculates the similarity between the feature amount of the extracted feature point in the template image and the feature amount of each feature point in the captured image (S52). The feature amount of each feature point is represented by a 128-dimensional feature vector, and the control unit 11 calculates the Euclidean distance between the two feature vectors as the similarity between the two feature amounts. The similarity of the feature amount is not limited to the Euclidean distance, but may be the Manhattan distance, the Hamming distance, or the like.

[0035] The control unit 11 identifies the feature point with the highest similarity among the similarities between the feature amounts of each feature point in the captured image calculated in step S52 (S53). The control unit 11 then determines whether the highest similarity is equal to or greater than a predetermined value (S54). If the control unit 11 determines that the highest similarity is not equal to or greater than the predetermined value (S54: NO), the control unit 11 excludes the feature point identified in step S53 from matching candidates for the feature points of the template image extracted in step S51 (S58). That is, the feature points of the template image extracted in step S51 are excluded from the matching process. This prevents inappropriate matching results by excluding a feature point in the captured image with the highest similarity to the feature points of the template image from matching candidates if the similarity is less than a predetermined value. Note that a smaller Euclidean distance indicates a higher similarity. Therefore, if the Euclidean distance is calculated in step S52, the control unit 11 identifies the feature point with the smallest Euclidean distance in step S53. Furthermore, in step S54, the control unit 11 determines whether the minimum Euclidean distance is equal to or less than a predetermined value, and if it determines that the minimum Euclidean distance is not equal to or less than the predetermined value (S54: NO), it excludes the feature points identified in step S53 from the matching candidates (S58).

[0036] If it is determined that the highest similarity is equal to or greater than a predetermined value (S54: YES), the control unit 11 identifies the feature point with the second highest similarity among the similarities to the feature amounts of the feature points in the captured image calculated in step S52 (S55). The control unit 11 then calculates the ratio of the second similarity to the highest similarity (S56) and determines whether the calculated ratio is equal to or greater than a predetermined ratio (a predetermined proportion) (S57). For example, the control unit 11 determines whether the calculated ratio is equal to or greater than 0.8 (80%). If it is determined that the ratio of the second similarity to the highest similarity is equal to or greater than the predetermined proportion (S57: YES), the control unit 11 excludes the feature point with the highest similarity identified in step S53 from matching candidates for the feature points of the template image extracted in step S51 (S58). As a result, if the similarity ratio (the ratio of the second similarity to the highest similarity) between the feature points in the captured image and the feature points in the template image that have the highest similarity is equal to or greater than a predetermined ratio, the feature points are excluded from the matching candidates. This means that two feature points in the captured image are similar to one feature point in the template image, and even if the feature points have the highest similarity, there is a possibility that an appropriate matching result will not be obtained. Therefore, if two feature points in the captured image are similar, excluding those two feature points from the matching candidates can prevent the occurrence of an inappropriate matching result. Note that if the Euclidean distance was calculated in step S52, the control unit 11 determines in step S57 whether the ratio of the second similarity to the highest similarity is, for example, 1.2 (120%) or less. If it is determined that the ratio is 1.2 or less (S57: YES), the feature points are excluded from the matching candidates (S58).

[0037] If it is determined that the ratio of the second similarity to the highest similarity is not equal to or greater than a predetermined ratio (S57: NO), the control unit 11 specifies the feature point with the highest similarity identified in step S53 as the feature point to be matched with the feature point of the template image extracted in step S51 (S59). After the process of step S58 or S59, the control unit 11 determines whether or not there are any feature points in the template image for which the above-mentioned process has not been performed (unprocessed feature points) (S60). If it is determined that there are unprocessed feature points (S60: YES), the control unit 11 returns to the process of step S51, extracts one of the unprocessed feature points in the template image (S51), and performs the processes of steps S52 to S59 for the extracted feature point.

[0038] The above-described process results in matching of feature points in the captured image with each feature point in the template image. Note that if the highest similarity to a feature point in the template image is less than a predetermined value, or if the ratio of the second-highest similarity to the highest similarity to a feature point in the template image is equal to or greater than a predetermined ratio, the feature point in the template image is excluded from matching. If it is determined that there are no unprocessed feature points (S60: NO), the control unit 11 returns to the process of FIG. 6. FIG. 9 shows an example of a feature point matching result. In FIG. 9, line segments connect the matched feature points in the template image and the feature points in the captured image. As shown in FIG. 9, it can be seen that the feature points in the captured image that correspond to the feature points in the template image are concentrated in the bill face area of ​​the captured image.

[0039] Based on the result of the feature point matching process, the control unit 11 calculates a perspective transformation matrix to be used when extracting the ID card photographed area (face area) from the captured image acquired from the user terminal 20 (S41). Specifically, the control unit 11 extracts feature points in the captured image that correspond to each feature point in the template image, and identifies a closed area surrounding the extracted feature points. As shown in FIG. 9, in the captured image, the feature points that correspond to each feature point in the template image are concentrated in the face area, so by identifying a closed area surrounding such feature points, the face area in the captured image can be identified. In the captured image shown on the right side of FIG. 9, the closed area indicated by the thick solid line is identified as the face area. Then, the control unit 11 calculates a perspective transformation matrix to convert the identified closed area (face area) into a rectangular area of ​​a predetermined shape and size, as shown in FIG. 10. FIG. 10 shows an example of the transformation process performed on the face area.

[0040] The control unit 11 determines whether keypoint matching was successful or not, for example, depending on whether the perspective transformation matrix could be calculated or not (S42), and if it determines that keypoint matching was not successful (S42: NO), specifically if the perspective transformation matrix could not be calculated, it sends information to the user terminal 20 indicating that reading the face area of ​​the card from the captured image failed (S43).

[0041] When the control unit 21 of the user terminal 20 receives information from the server 10 indicating that reading of the face area has failed, it displays a message such as that shown in FIG. 8B on the display unit 25 to notify the user to re-photograph the ID card (S44). The screen shown in FIG. 8B notifies the user that re-photographing is necessary because the face area could not be identified from the captured image, and has an OK button for instructing the user to re-photograph. On the screen shown in FIG. 8B, the user confirms that re-photographing is necessary and presses the OK button. When the control unit 21 receives an operation on the OK button on the screen shown in FIG. 8B, it returns to the processing of step S31, and when it determines that an instruction to execute the photographing process has been received (S31: YES), it repeats the processing from step S32 onwards.

[0042] If it is determined that keypoint matching was successful (S42: YES), specifically, if the perspective transformation matrix was calculated, the control unit 11 (extraction unit) performs a matrix operation using the calculated perspective transformation matrix on the closed region (face region) identified in the captured image, and extracts a face region that has been converted (corrected) into a rectangle (S45). As a result, even if the image does not show the ID card from the front, as shown in FIG. 10, it is possible to extract the rectangular face region as if it were captured from the front. After extracting the face region, the control unit 11 acquires personal information (face information) included in the face region as needed (S46). For example, by setting the position of the printing region for each piece of personal information included in the face region using coordinate values ​​(x, y) with the upper left corner of the face region as the origin, the X axis extends rightward from the origin, and the Y axis extends downward, the control unit 11 can acquire each piece of personal information from each printing region in the face region. The control unit 11 can read the personal information from each printing region using, for example, OCR (Optical Character Recognition).

[0043] The control unit 11 ends the series of processes after the processing of step S43 or the processing of step S46. If the server 10 is a server that performs processing to accept procedures from the user terminal 20, the control unit 11 performs processing to register the card information (personal information) acquired in step S46 as registration information in a database or the like that has been prepared in advance.

[0044] In the above-described keypoint matching process using a template image, the personal information contained in the template image is masked, and the masked area is excluded from the keypoint matching process. Because personal information is text information that differs for each ID card owner, excluding the personal information from the keypoint matching process improves the accuracy of the matching process. By performing keypoint matching using such a template image, it is possible to properly extract the card face area from a captured image, even if the captured image shows light reflected by the hologram when photographing an ID card with a hologram on its surface, or even if the image shows an ID card photographed against a background of text-heavy printed material such as a flyer or leaflet.

[0045] In this embodiment, the processes shown in Fig. 6 that are performed by the server 10 may be configured to be performed locally by the user terminal 20. For example, the server 10 may download a card face extraction program incorporating the template image generated by the process shown in Fig. 3 to the storage unit 22 of the user terminal 20, thereby allowing all of the processes shown in Fig. 6 to be executed by the user terminal 20. In this case, the control unit 21 of the user terminal 20 performs the processes of steps S31 to S34 in Fig. 6, as well as steps S36 to S42 and S45 to S46 that were performed by the server 10 in Fig. 6. In other words, the user terminal 20 may be configured to extract the face area of ​​the ID card being photographed from the captured image by keypoint matching using the template image.

[0046] The configuration of this embodiment may also be incorporated into an image reading device such as a scanner. That is, the image reading device may be equipped with a function for extracting a card face area from an image obtained by reading the face of an ID card (an image corresponding to a photographed image) by keypoint matching using a template image. Furthermore, a scanner having such a function may further be equipped with a configuration for acquiring personal information from the card face area extracted from the read image.

[0047] In this embodiment, the accuracy of keypoint matching can be improved by masking the printed area of ​​personal information on the template image used for keypoint matching. Therefore, even when an ID card is photographed using a camera with simple capabilities, such as a smartphone or mobile phone camera, the ID card face area can be accurately extracted. Furthermore, in this embodiment, matching with the template image is performed using keypoint matching with features such as SIFT features, improving processing accuracy. For example, the ID card face area can be accurately extracted from the photographed image regardless of the lighting conditions, background pattern, magnification or reduction ratio, ID card orientation, etc., or even if the ID card face has deteriorated or been deformed over time. This makes it possible to extract the face area even from photographed images where extraction was difficult using conventional techniques, such as when the photographed image contains shadows, light reflections due to holograms, complex background patterns, is photographed at a magnification or reduction, or is not photographed from the front of the ID card.

[0048] In each device equipped with the configuration of the present embodiment described above, if the face area of ​​an ID card can be extracted from a captured image, it becomes possible to determine the authenticity of the ID card in the captured image (the ID card being photographed) and determine that it is a valid ID card. By reading personal information from the captured image (face area) determined to be a valid ID card in this way using, for example, OCR, it becomes possible to perform identity verification (identity authentication) or register personal information.

[0049] The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0050] 10 Servers 11 Control section 12 Storage section 13 Communications Department 20 User terminal 21 Control section 22 Memory section 23 Communications Department

Claims

1. Acquire a photograph of the card, Acquire feature amounts of a plurality of feature points in the acquired photographed image; calculating a similarity between the feature amount of each feature point in a template image obtained by masking an area of ​​the personal information printed on a card, the area of ​​the personal information being cut out from a photographed image of the card, and the feature amount of each feature point in the acquired photographed image; determining whether the second highest similarity calculated for each feature point in the template image is equal to or greater than a predetermined ratio with respect to the highest similarity; If it is determined that the similarity is equal to or greater than the predetermined ratio, the feature points in the captured image for which the highest similarity is calculated are not matched with the feature points in the template image; if it is determined that the similarity is not equal to or greater than the predetermined ratio, the feature points in the captured image for which the highest similarity is calculated are matched with the feature points in the template image; Based on the matching result, the photographed area of ​​the card in the acquired photographed image is extracted. A program that causes a computer to perform a process.

2. The template image is generated by accepting a photographed area of ​​the card and an area of ​​the personal information included in the photographed area from a photographed image of a card on which the personal information is printed, cutting out the accepted photographed area of ​​the card from the photographed image, and masking the accepted area of ​​the personal information from the cut-out photographed area. The program according to claim 1.

3. The acquired image is subjected to noise reduction processing, Extracting the photographed area of ​​the card from the noise-removed photographed image.

3. The program according to claim 1, which causes the computer to execute processing.

4. Downsampling is performed on the captured image, Extracting the photographed area of ​​the card from the downsampled photographed image 4. The program according to claim 1, which causes the computer to execute a process.

5. calculating a transformation matrix for transforming the photographed area of ​​the card in the photographed image into a predetermined shape and size; A transformation process is performed on the photographed area of ​​the card in the photographed image based on the calculated transformation matrix to extract the photographed area of ​​the card.

5. The program according to claim 1, which causes the computer to execute a process.

6. Acquire a photograph of the card, Acquire feature amounts of a plurality of feature points in the acquired photographed image; calculating a similarity between the feature amount of each feature point in a template image obtained by masking an area of ​​the personal information printed on a card, the area of ​​the personal information being cut out from a photographed image of the card, and the feature amount of each feature point in the acquired photographed image; determining whether the second highest similarity calculated for each feature point in the template image is equal to or greater than a predetermined ratio with respect to the highest similarity; If it is determined that the similarity is equal to or greater than the predetermined ratio, the feature points in the captured image for which the highest similarity is calculated are not matched with the feature points in the template image; if it is determined that the similarity is not equal to or greater than the predetermined ratio, the feature points in the captured image for which the highest similarity is calculated are matched with the feature points in the template image; Based on the matching result, the photographed area of ​​the card in the acquired photographed image is extracted. An information processing method in which processing is performed by a computer.

7. An information processing device having a control unit, The control unit Acquire a photograph of the card, Acquire feature amounts of a plurality of feature points in the acquired photographed image; calculating a similarity between the feature amount of each feature point in a template image obtained by masking an area of ​​the personal information printed on a card, the area of ​​the personal information being cut out from a photographed image of the card, and the feature amount of each feature point in the acquired photographed image; determining whether the second highest similarity calculated for each feature point in the template image is equal to or greater than a predetermined ratio with respect to the highest similarity; If it is determined that the similarity is equal to or greater than the predetermined ratio, the feature points in the captured image for which the highest similarity is calculated are not matched with the feature points in the template image; if it is determined that the similarity is not equal to or greater than the predetermined ratio, the feature points in the captured image for which the highest similarity is calculated are matched with the feature points in the template image; Based on the matching result, the photographed area of ​​the card in the acquired photographed image is extracted. Information processing device.

Images