Learning program, learning method, learning device, authenticity determination program, authenticity determination method, and authenticity determination device

Abstract

Data is prepared that can be used to train an authenticity determination model that determines the authenticity of card media that have nearly identical appearances and patterns. [Solution] The learning program causes a computer to execute the following steps: an extraction procedure that extracts an image array contained in video image data captured in chronological order from a card medium as an initial image array; a duplication procedure that generates an array of static images by duplicating one image selected from the initial image array any number of times; an expansion procedure that copies the static image array and applies a predetermined expansion process to any static image contained in the copied static image array to generate an array of dynamic images; and an authenticity determination learning procedure that trains an authenticity determination model that has been loaded with the initial image array, static image array, and dynamic image array to learn the card medium, and records the trained authenticity determination model in a learning model recording unit.

Description

[Technical Field]

[0001] The present invention relates to a learning program, a learning method, a learning device, an authenticity determination program, an authenticity determination method, and an authenticity determination device. [Background technology]

[0002] Various techniques have been devised to determine the authenticity of cards used in trading card games. For example, a technique is known in which a near-field communication (NFC) chip storing information for authenticity determination is embedded in a card and the information is read by an NFC tag reader to determine the authenticity of the card. This technique makes it possible to determine the authenticity of a card with high accuracy. However, because the manufacturing cost of NFC chips is high, incorporating an NFC chip into a card increases the manufacturing cost of the card.

[0003] For this reason, attempts have been made to photograph the card with a camera built into a mobile device such as a smartphone and use the photographed image to verify the authenticity of the card. In addition to verifying the authenticity of the card, attempts have also been made to perform authentication processing (also called check-in) using the photographed image in an access-managed authentication system.

[0004] For example, Patent Document 1 discloses a technology in which a guide for changing the shooting angle of a moving image of a card medium captured by a camera is displayed on the screen of a terminal device, the display position of the guide is changed at least once, and the authenticity of a card medium is determined based on time series information of reflected light from a specific surface of the card medium contained in the moving image data of the card medium captured by the camera while the guide is displayed. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 7594140 Summary of the Invention [Problem to be solved by the invention]

[0006] Because the problem of authenticating cards involves binary classification between images with very similar features, developing an effective learning method for an authentication model is challenging. In particular, when authenticating commercial products, the difficulty of comprehensively collecting negative (counterfeit) samples makes it difficult to achieve sufficient accuracy using conventional machine learning methods. For example, conventional authentication models use lightweight models (based on convolutional neural networks (CNNs)) that can run on edge devices such as smartphones. However, these models are prone to false positives for some counterfeit patterns, resulting in unstable classification accuracy. In particular, authentication models often fail to properly extract useful features that affect classification accuracy (e.g., light scattering characteristics resulting from hologram processing on card media). In such cases, conventional authentication models mistakenly identify counterfeit cards as genuine. Therefore, there is a need for an authentication model that can efficiently learn the information necessary for image classification.

[0007] The technology disclosed in the aforementioned Patent Document 1 does not take into consideration the efficient learning of information necessary for image classification by an authentication determination model. Therefore, preparing the data necessary for training the authentication determination model required a great deal of effort. Thus, in the past, it was difficult to ensure sufficient classification accuracy in cases such as authentication determination, where "features between different classes are similar." Particularly in authenticating card media, there are many cases where the appearance and pattern of card media are nearly identical, but only some subtle features or physical properties differ. In other words, because the visual differences between different classes (genuine and counterfeit) are small, the nature of this problem differs from conventional image classification problems such as those used to distinguish people or automobiles. Furthermore, even when a general image classification model is used to authenticate card media, the amount of data necessary for training is insufficient, making it impossible to determine the authenticity of card media with sufficient accuracy.

[0008] The present invention has been made in view of the above circumstances, and aims to provide data from which an authenticity determination model for determining the authenticity of card media can be trained. [Means for solving the problem]

[0009] A learning program according to the present invention studies a card medium whose predetermined surface has been subjected to processing that changes the reflection state of light reflected from the predetermined surface of the card medium. The learning program causes a computer to execute the following steps: an extraction procedure for extracting an arbitrary number of video data items from multiple video data items captured in time series of the card medium, and for each extracted video data item, extracting an image array contained in the video data as an initial image array; a duplication procedure for duplicating an image selected from the initial image array an arbitrary number of times to generate an array of static images; an extension procedure for copying the static image array and performing a predetermined extension process on any static image contained in the copied static image array to generate a dynamic image array, and recording the static image array and the dynamic image array in a learning data recording unit as authenticity determination learning data for learning the card medium; and an authenticity determination learning procedure for training the card medium using an authenticity determination model that has been loaded with the initial image array, the static image array, and the dynamic image array, and recording the trained authenticity determination model in a learning model recording unit. The above learning program is one aspect of the present invention, and a learning method and a learning device that reflect one aspect of the present invention are configured in the same manner as the above authenticity determination program.

[0010] The authenticity determination program of the present invention determines the authenticity of a card medium whose predetermined surface has been subjected to processing that changes the reflection state of light reflected from the predetermined surface. This authenticity determination program includes a display control procedure that displays a guide on the screen of the terminal device to change the shooting angle of a moving image of the card medium captured by a camera provided in the terminal device and changes the display position of the guide during the shooting period, a known image detection procedure that detects a known image formed on the card medium from the moving image data of the card medium captured according to the guide, and an extraction procedure that, when a known image is detected, uses as input second moving image data of the card medium captured by the camera with the guide displayed and reads out from the recording unit of the terminal device, extracts an arbitrary number of moving image data from multiple moving image data captured in chronological order of the card medium, and for each extracted moving image data, extracts the arrangement of images contained in the moving image data as an initial image arrangement, and The computer is caused to execute a learning program that causes a computer to execute a duplication procedure in which one image selected from an image array is duplicated any number of times to generate an array of static images, an extension procedure in which the static image array is copied and any static image included in the copied static image array is subjected to a predetermined extension process to generate an array of dynamic images, and the static image array and the dynamic image array are recorded in a learning data recording unit as authenticity determination learning data for learning the card medium, and an authenticity determination learning procedure in which the card medium is trained into an authenticity determination model that has been loaded with the initial image array, the static image array, and the dynamic image array, and the computer is caused to execute a determination procedure in which the authenticity of the card medium is determined based on time series information of reflected light from a predetermined surface of the card medium using a trained authenticity determination model that has trained the card medium. The above-mentioned authenticity determination program is one aspect of the present invention, and an authenticity determination program, an authenticity determination method, and an authenticity determination device that reflect one aspect of the present invention are configured in the same manner as the above-mentioned authenticity determination program. [Effects of the Invention]

[0011] According to the present invention, data that can be learned by an authenticity determination model that determines the authenticity of a card medium is prepared, and the authenticity determination model can be trained. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is an overall configuration diagram showing an overview of an authenticity determination system according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing an example of the hardware configuration of each of a server and a terminal device that constitute an authenticity determination system according to a first embodiment of the present invention. [Figure 3] 1 is a block diagram showing the functional configuration of an authenticity determination system according to a first embodiment of the present invention. [Figure 4] 1 is a block diagram showing an example of the internal configuration of a learning device according to a first embodiment of the present invention. [Figure 5] 3 is a flowchart showing an example of a known item learning process according to the first embodiment of the present invention. [Figure 6] 4 is a flowchart showing an example of an authenticity determination learning process according to the first embodiment of the present invention. [Figure 7] 1 is a flowchart illustrating an example of a training data extension process according to the first embodiment of the present invention. [Figure 8] 3A to 3C are diagrams showing examples of images from which known item learning data is generated from video data according to the first embodiment of the present invention. [Figure 9] 5A to 5C are diagrams illustrating an example of a guide frame display based on a guide frame display process performed by a display processing unit according to the first embodiment of the present invention. [Figure 10] 5A to 5C are diagrams showing examples of changes in the angle at which a card is photographed by a camera according to the first embodiment of the present invention. [Figure 11] 1A and 1B are diagrams showing examples of images in which whiteout appears according to the first embodiment of the present invention. [Figure 12] 3 is a flowchart showing an example of the procedure of an authenticity determination process performed by the authenticity determination system according to the first embodiment of the present invention. [Figure 13]10 is a flowchart showing an example of a procedure for a guide frame display process by a display processing unit according to the first embodiment of the present invention. [Figure 14] 5 is a flowchart showing an example of a procedure for shooting determination processing by a shooting determination unit according to the first embodiment of the present invention. [Figure 15] 5 is a flowchart showing an example of a procedure for a known item detection process performed by a known item detection unit according to the first embodiment of the present invention. [Figure 16] 10 is a flowchart showing an example of the procedure of two-stage blown-out highlight detection processing by a two-stage blown-out highlight detection unit according to the first embodiment of the present invention. [Figure 17] 5 is a flowchart showing an example of the procedure of an authenticity determination process performed by an authenticity determination unit according to the first embodiment of the present invention. [Figure 18] FIG. 10 is a block diagram showing an example of the hardware configuration of each of a server and a terminal device that constitute an authenticity determination system according to a second embodiment of the present invention. [Figure 19] FIG. 10 is a block diagram showing the functional configuration of an authenticity determination system according to a second embodiment of the present invention. [Figure 20] FIG. 10 is a block diagram showing the functional configuration of an authenticity determination system according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted.

[0014] [First embodiment] <Outline of the authenticity determination system> FIG. 1 is a diagram showing the overall configuration of an authentication system 100 according to a first embodiment of the present invention. The authenticity determination system 100 is a system that determines whether, for example, cards (an example of card media) used in a trading card game are genuine (authentic) or counterfeit (non-genuine). The authenticity determination system 100 includes a learning device 3, a smartphone 21, and a PC (Personal Computer) 22. The smartphone 21 and PC 22 can be connected to the learning device 3 via a network N such as the Internet. In the following description, the smartphone 21 and PC 22 are collectively referred to as the information processing terminal 2. Note that a dedicated gaming terminal may also be used as the information processing terminal 2.

[0015] The card Cd (see FIG. 9) that is the subject of authentication determination by the authentication determination system 100 is a card that has a hologram and embossing (an example of textured processing) on ​​its surface (an example of a predetermined surface), and may become glossy in a photographing environment with a light source such as a lamp. The information processing terminal 2 determines the authenticity of the card based on information obtained by photographing the card with the camera 116 (see FIG. 2).

[0016] The authenticity determination system 100 according to this embodiment uses information such as information about changes in reflected light from the card according to the shooting angle of the camera 116 of the information processing terminal 2 as information obtained by the camera 116 photographing the card. Furthermore, the information processing terminal 2 according to this embodiment changes the pattern of changes over time in the reflected light from the card by displaying on the screen of the display device 119 a guide frame that prompts the user to move the terminal device in various directions such as up, down, left, and right while photographing the card, i.e., a guide frame for changing the shooting angle of the card by the camera 116.

[0017] However, light reflected from the card can cause defective photographs, such as overexposure, in the captured image, resulting in overexposure. This can easily lead to defective photographs, such as overexposure (hereafter referred to as "overexposure"), which can affect authentication. Overexposure is a phenomenon in which excessive light is used during photography, resulting in the loss of color and gradation information, making part or all of the image appear pure white. When overexposure occurs, the shape and texture information contained in the overexposed area becomes saturated and lost, significantly reducing the accuracy of authenticity determination and check-in processing. Therefore, there is a need for a system that can automatically detect overexposure and warn the user.

[0018] Although excessive light incident on a camera's image sensor or the like can cause photon saturation and result in overexposure, processing can be performed to reduce the effect of photon saturation on the captured image by changing the dynamic range, etc. On the other hand, even if excessive light is not incident on a camera's image sensor or the like, overexposure can occur in the captured image due to light reflection from the surface of the subject, etc.

[0019] However, the software built into mobile devices alone cannot perform simple processing such as determining areas of a captured image that have been blown out due to the effects of auto exposure (AE). Furthermore, mobile devices also incorporate software that processes captured images, and captured images after automatic correction do not contain data before development, such as RAW data. Therefore, authenticity of card media must be determined based on captured images that have been developed by the software or hardware built into the mobile device. For this reason, a warning message is displayed instructing the user to re-take a photo of the card to prevent overexposure.

[0020] Then, information processing terminal 2 performs photography determination processing, known item detection processing, two-stage blown-out highlight detection processing, normalization processing, and authenticity determination processing. In the photography determination processing, information processing terminal 2 determines whether camera 116 of information processing terminal 2 is photographing a physical card medium, using as a key whether the movement vector of 6-axis sensor 117 (see FIG. 2) provided in information processing terminal 2 and the movement vector of feature points extracted from the image photographed by camera 116 change simultaneously and in the same direction.

[0021] Furthermore, in a known item detection process, the information processing terminal 2 determines whether or not the subject is a known item (for example, a card) based on the captured image. Furthermore, in a two-stage blown-out highlight detection process, the information processing terminal 2 determines whether or not the captured image has blown-out highlights that would prevent accurate authenticity determination. Furthermore, in a normalization process, the information processing terminal 2 normalizes a captured image that is determined not to have blown-out highlights.

[0022] Furthermore, in the authenticity determination process, the information processing terminal 2 determines the authenticity of the card based on information on the change in the time direction of the temporal change pattern of the reflected light from the card, which is included in the video recorded when the photographing determination is performed, i.e., based on the time series information of the reflected light from the card. The information processing terminal 2 performs the authenticity determination process using an authenticity determination model 202b (see FIG. 3, which will be described later) that has learned the authenticity of the card medium using authenticity determination learning data 40c (see FIG. 4, which will be described later) expanded by data expansion according to this embodiment.

[0023] The learning device 3 is a server that manages a master image used for photographic determination by the information processing terminal 2, a known item detection model 45a used for known item detection, and an authenticity determination model 45b used for authenticity determination (see FIG. 3 for all of these). The learning device 3 synchronizes the master image stored in the information processing terminal 2 with the master image managed by the learning device 3 by transmitting the master image to the information processing terminal 2 via the network N at predetermined intervals. The learning device 3 also synchronizes the known item detection model 45a and the authenticity determination model 45b stored in the information processing terminal 2 with the master image managed by the learning device 3 by transmitting the known item detection model 45a and the authenticity determination model 45b to the information processing terminal 2 via the network N at predetermined intervals.

[0024] The information processing terminal 2 is an example of an authenticity determination device. An application (not shown) including a program for executing each process related to authenticity determination according to this embodiment is stored in the nonvolatile storage 121 (see FIG. 2) of the information processing terminal 2. The control unit 110 (see FIG. 2) of the information processing terminal 2 starts the application based on a user operation, and performs a predetermined process by reading the program recorded in the nonvolatile storage 121.

[0025] The predetermined processing includes the above-mentioned processing of photograph determination, authenticity determination, etc. The predetermined processing also includes processing in which control unit 110 of information processing terminal 2 displays, as a guide, an image of a guide frame (hereinafter also simply referred to as "guide frame") that specifies the display position of the card on the screen of display device 119 on the screen of display device 119. Furthermore, the predetermined processing includes processing in which information captured by camera 116 while the user is performing the action of aligning the display position of the card with the guide frame is recorded as a moving image.

[0026] The input device 118 generates an operation signal according to an operation input by the user, and supplies the operation signal to the control unit 110 . The display device 119 displays a guide frame and the like on the screen based on the control of the control unit 110.

[0027] <Example of hardware configuration for authenticity determination system> Next, an example of the hardware configuration of the authenticity determination system 100 according to the first embodiment of the present invention will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the learning device 3 and the information processing terminal 2 that constitute the authenticity determination system 100.

[0028] (Example of learning device configuration) The learning device 3 is an example of a calculator that operates as a computer. The learning device 3 includes a control unit 130, a nonvolatile storage 135, and a network interface 136, which are all connected to a bus B1.

[0029] The control unit 130 includes a central processing unit (CPU) 131, a graphics processing unit (GPU) 132, a read only memory (ROM) 133, and a random access memory (RAM) 134. The control unit 130 may be configured using a field programmable gate array (FPGA).

[0030] The CPU 131 reads out program code of software that realizes each function according to this embodiment from the ROM 133, loads it into the RAM 134, and executes it. Variables, parameters, etc. that are generated during the calculation processing of the CPU 131 are temporarily written to the RAM 134. These variables, parameters, etc. written to the RAM 134 are read out by the CPU 131 as appropriate. The GPU 132 performs calculations required when performing various learning processes according to this embodiment.

[0031] Examples of nonvolatile storage 135 include a hard disk drive (HDD), a solid state drive (SSD), an optical disk, a magneto-optical disk, and nonvolatile memory. In addition to an operating system (OS) and various parameters, this nonvolatile storage 135 also stores programs for running the learning device 3. The programs for running the learning device 3 may be stored in ROM 133. In other words, ROM 133 and nonvolatile storage 135 are used as an example of a computer-readable, non-transitory recording medium that stores a program executed by the learning device 3.

[0032] The nonvolatile storage 135 also includes a learning data database (DB) 40 that stores learning data used in machine learning, and a master image DB 50 (see FIG. 3) that stores master images. The nonvolatile storage 135 also includes a server-side learning model DB 45 (see FIG. 3) that stores a known item detection model 202a and an authenticity determination model 202b.

[0033] For example, a network interface card (NIC) or the like is used as the network interface 136. The network interface 136 transmits and receives various data to and from the information processing terminal 2 via a dedicated line or the like connected to a terminal of the NIC and via the network N.

[0034] (Example of terminal device configuration) The information processing terminal 2 is an example of a calculator that operates as a computer capable of executing various programs. The information processing terminal 2 includes a control unit 110, a network interface 115, a camera 116, a six-axis sensor 117, an input device 118, a display device 119, an input / output interface 120, and a non-volatile storage 121, which are all connected to a bus B2.

[0035] The control unit 110 includes a CPU 111, a GPU 112, a ROM 113, and a RAM 114. The control unit 110 may be configured as an FPGA.

[0036] The CPU 111 reads out program code of software that realizes each function according to the present embodiment from the ROM 113, loads it into the RAM 114, and executes it. Variables, parameters, etc. that are generated during the arithmetic processing of the CPU 111 are temporarily written to the RAM 114. These variables, parameters, etc. written to the RAM 114 are read out as appropriate by the CPU 111. The CPU 111 performs processes such as processing of the OS in the information processing terminal 2 and management of input and output of data performed by each unit in the information processing terminal 2. The GPU 112 performs calculations required for performing various inference processes according to this embodiment in addition to rendering images.

[0037] For example, a NIC or the like is used as the network interface 115. The network interface 115 can acquire various data from the learning device 3 via a LAN, a dedicated line, or the like connected to a terminal of the NIC, and can communicate with other information processing terminals 2.

[0038] The camera 116 includes an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and converts light from a subject focused on a lens (not shown) into image data. The camera 116 according to this embodiment is capable of capturing not only still images but also moving images.

[0039] The six-axis sensor 117 is an inertial sensor that can detect six-axis inertial forces, including accelerations in the three axes (front-back, left-right, and up-down) and angular velocities in the three axes that are applied to the information processing terminal 2. Information on the values ​​detected by the six-axis sensor 117 is referenced when performing the above-described photographing determination. Note that the inertial sensor included in the information processing terminal 2 is not limited to the six-axis sensor 117, and may be any other inertial sensor implemented in the information processing terminal 2.

[0040] The input device 118 is configured by a touch sensor on a touch panel in the smartphone 21 (see FIG. 1), and is configured by a mouse, keyboard, and the like in the PC 22. The display device 119 is configured by an LCD (Liquid Crystal Display), an organic EL (Electro Luminescence) display, or the like.

[0041] The input / output interface 120 converts operation signals received from the input device 118 into data in a predetermined format and passes the data to the CPU 111. The input / output interface 120 also passes images of guide frames and the like output from the control unit 110 to the display device 119. The display device 119 may be equipped with a speaker (not shown), in which case the input / output interface 120 can also output audio signals to the speaker.

[0042] The nonvolatile storage 121 may be, for example, an HDD, SSD, optical disk, magneto-optical disk, or nonvolatile memory. In addition to the OS and various parameters, the nonvolatile storage 121 stores programs for operating the information processing terminal 2, a local image DB 201 based on the master image DB 50 transmitted from the learning device 3, a known item detection model 202a, and an authenticity determination model 202b (see FIG. 3 ).

[0043] A program for causing the learning device 3 to function may be stored in the ROM 113. In other words, the ROM 113 and the nonvolatile storage 121 are used as an example of a computer-readable non-transitory recording medium that stores a program executed by the information processing terminal 2.

[0044] If the nonvolatile storage 121 is an external recording medium such as an optical disc, an optical drive (not shown) reads the nonvolatile storage 121 and extracts data from it. The data extracted by the optical drive is used for predetermined arithmetic processing by the CPU 111.

[0045] <Example of functional configuration of authenticity determination system> Next, the functional configuration of the authenticity determination system 100 will be described with reference to FIG. Fig. 3 is a block diagram showing the functional configuration of the authenticity determination system 100. In Fig. 3, the communication network N is omitted.

[0046] (Server functional configuration example) As shown in Fig. 3, the learning device 3 includes a learning data DB 40, a master image DB 50, and a server-side learning model DB 45, which are configured in the non-volatile storage 121 shown in Fig. 2. The learning data DB 40 is an example of a learning data recording unit that records learning data for each learning model stored in the server-side learning model DB 45. The master image DB 50 is a DB that serves as the basis for the local image DB 201 that stores images referenced by the photography determination unit 206 of the information processing terminal 2 when performing photography determination, and stores photographed images of all types of legitimate (authentic) cards.

[0047] The server-side learning model DB 45 is an example of a learning model recording unit that records a known item detection model 45a and a trained authenticity determination model 45b constructed by a neural network that performs learning taking time axis information into consideration. The known item detection model 45a and the authenticity determination model 45b are composed of weight data whose weights have been adjusted by learning.

[0048] The known item detection model 45a is an example of AI (Artificial Intelligence) that is based on, for example, a YOLO-based high-speed multitasking object detection network and can simultaneously estimate the identifier of a card for a card game and the written language (Japanese, English, Simplified Chinese, etc.). The known item detection model 45a can be automatically learned and updated by the learning device 3 even when a new type of card is added.

[0049] The authenticity determination model 45b is an image recognition model, such as SlowFast, that analyzes the temporal change patterns of reflected light from diffractive optical structures, such as holograms and embossments, on the card surface and classifies these temporal change patterns based on learning to determine the authenticity of the card medium. Depending on the processing applied to a specific surface of the card medium, the temporal change patterns of reflected light from each specific surface corresponding to incident light from multiple different angles of incidence are different. Therefore, the input to the authenticity determination model 45b is a short video captured by a user while slowly tilting their smartphone up, down, left, and right. The authenticity determination model 45b continuously acquires brightness changes that vary for each shooting angle due to the movement during shooting.

[0050] For neural networks that perform learning taking time axis information into account, spatiotemporal architectures such as 3D-CNN (3-Dimensional Convolutional Neural Network), SlowFast, and TimeSformer (Time-Space Transformer) can be applied. The authenticity determination model 45b comprehensively maps the spatial features of each frame of the input video image and the temporal changes between frames into a feature space. During inference for authenticity determination, the authenticity determination model 45b inputs this spatiotemporal feature sequence into a transfer learning model that performs binary classification of authenticity. The authenticity determination model 45b determines the card as "authentic" only if the confidence score exceeds a predetermined threshold. Therefore, the authenticity determination model 45b can identify the authenticity of gaming cards with extremely high accuracy using only optical features, without embedding semiconductor elements such as NFC in the card.

[0051] For example, when the known item detection model 45a and the authenticity determination model 45b are constructed using a deep learning framework called PyTorch, the known item detection model 45a and the authenticity determination model 45b are implemented in the PTH file format. Note that the file formats of the known item detection model 45a and the authenticity determination model 45b may be other file formats such as the GGML file format or the ONNX (Open Neural Network eXchange) file format.

[0052] The learning device 3 periodically transmits the master image stored in the master image DB 50, and the known item detection model 45a and the authenticity determination model 45b stored in the server-side learning model DB 45 to the information processing terminal 2. The known item detection model 45a and the authenticity determination model 45b received by the information processing terminal 2 are stored in the local-side inference model DB 202 as the known item detection model 202a and the authenticity determination model 202b, respectively.

[0053] (Example of terminal device functional configuration) The information processing terminal 2 includes a local image DB 201, a local inference model DB 202, a display processing unit 203, a video image capturing unit 204, a card area detection unit 205, a six-axis sensor 117, a capturing judgment unit 206, a known item detection unit 207, a two-stage whiteout detection unit 208, a normalization unit 209, and an authenticity judgment unit 210.

[0054] The local image DB 201 is the master image DB 50 transmitted from the learning device 3 in a format that can be accessed (referenced) by an application of the information processing terminal 2.

[0055] The local-side inference model DB 202 is a database that stores the known item detection model 202a and the authenticity determination model 202b transmitted from the learning device 3. As described above, the known item detection model 202a and the authenticity determination model 202b have the same contents as the known item detection model 45a and the authenticity determination model 45b stored in the server-side learning model DB 45, but have different symbols.

[0056] The learning data for the authenticity determination model 202b is time-series information on the light reflected from the surface of a card, which is included in video data of a genuine (authentic) card photographed by the camera 116 based on guidance from the guide frame. The authenticity determination model 202b then inputs video data of the card to be authenticated, photographed based on guidance from the guide frame, and outputs an authenticity determination result for the card. The authenticity determination model 202b analyzes the pattern of change over time in the light reflected from the card to be authenticated using, for example, a classifier that uses SlowFast, and determines the authenticity of the card by learning-based classification based on whether the pattern of change over time matches the characteristic movement seen in genuine cards.

[0057] The display processing unit 203 (an example of a display control unit) is a processing unit that displays a guide frame on the screen of the display device 119 (see FIG. 2). The guide frame is indicated, for example, by a thick red frame. When the display processing unit 203 detects that the frame of the card has fit within the guide frame, it displays a sign such as "OK" on the screen. Thereafter, the display processing unit 203 moves the guide frame to another position on the screen and prompts the user to perform an operation to fit the frame of the card into that position. Examples of display of the guide frame by the display processing unit 203 will be described in detail with reference to FIGS. 9 and 10 below.

[0058] The video capturing unit 204 encodes the image data (an example of second video data) obtained from the camera 116 as a video at a predetermined frame rate. The video capturing unit 204 saves the captured video in the non-volatile storage 121 in a general-purpose format such as H.264 / MP4. The video capturing unit 204 continues capturing video while the display processing unit 203 is changing the display position of the guide frame.

[0059] The card area detection unit 205 detects the position and type of the card from image data obtained from the camera 116, for example, using YOLO (You Only Look Once). YOLO is a real-time object detection model that processes an image all at once and simultaneously estimates the position and category of an object at high speed. After determining the type of card using YOLO, the card area detection unit 205 detects the card area using, for example, KAZE. However, the card area detection unit 205 may also use YOLO to perform both the detection of the card type and the card area. Alternatively, the card area detection unit 205 may also use AKAZE (Accelerated-KAZE) to perform both the detection of the card type and the card area. By using such a technique, sufficient processing speed can be maintained even when the information processing terminal 2 is a smartphone 21.

[0060] The photography determination unit 206 determines whether the camera 116 of the information processing terminal 2 is photographing a physical card medium whose authenticity is to be determined, based on the movement of the six-axis sensor 117 (see FIG. 2) and the movement of the card position detected by the card area detection unit 205. Specifically, the photography determination unit 206 executes a photography determination procedure to determine whether the camera 116 is photographing a physical card medium by comparing the movement vector of the card position extracted from the video data transmitted from the camera 116 with each of the movement vectors in the multiple coordinate axis directions acquired from the six-axis sensor 117. Through this photography determination procedure, the photography determination unit 206 can determine whether the movement vector of the card position extracted from the video data and the movement vector acquired from the six-axis sensor 117 are moving simultaneously in the same direction.

[0061] By performing the photographing judgment by the photographing judgment unit 206, even if a cheating method (unauthorized operation) related to the method of photographing a card is executed, the cheating method can be detected. Examples of cheating methods related to the method of photographing a card include a method of photographing a moving image obtained by photographing the card in advance, playing it in front of the camera 116 of the information processing terminal 2, and a method of photographing a high-resolution copy image of the card with the camera 116.

[0062] Furthermore, the shooting determination unit 206 generates coordinate information of the guide frame for changing the guide frame to the next display position during execution of the shooting determination, and outputs the information to the display processing unit 203 .

[0063] The known item detection unit 207 uses the known item detection model 202a to execute a known image detection procedure that detects a known image formed on a card medium that has been processed on a predetermined surface from video image data of the card medium that has been photographed according to the guide. In this embodiment, a known item is an example of a known image (genuine image), and the known item detection unit 207 is an example of a known image detection unit. The known item detection unit 207 estimates information unique to the card medium that is included in the image of the card medium based on the known image, and outputs a certainty score that indicates the certainty of the estimated unique information.

[0064] The two-stage overexposure detection unit 208 is activated only when the certainty score of the subject area output by the known item detection model 202a of the known item detection unit 207 is neither too low nor too high, but is in the middle range. This is because the design takes into consideration that overexposure can occur frequently even in well-intentioned usage environments, but has characteristics that differ from intentional concealment.

[0065] In the procedure for detecting poorly photographed areas executed by the two-stage blown-out highlight detection unit 208, when it is not certain that a known image has been detected, the two-stage blown-out highlight detection unit 208 detects, for example, the presence or absence of blown-out highlights in the image as a poorly photographed area in the image on the card medium. Note that when the certainty score is less than a predetermined certainty threshold, the two-stage blown-out highlight detection unit 208 determines that it is not certain that the photographed subject is a known image. Here, the two-stage blown-out highlight detection unit 208 detects the presence or absence of blown-out highlights in two stages: global analysis and local analysis. Details of the global analysis and local analysis will be described later with reference to FIG. 16.

[0066] This two-stage blown-out highlight detection unit 208 is an example of a poorly captured part detection unit that, when it detects that an image has blown-out highlights, issues an instruction to shoot a moving image on the card medium again and operates the display processing unit 203. The two-stage blown-out highlight detection unit 208 issues an instruction to shoot a moving image on the card medium again by outputting, to the display processing unit 203, information instructing the display processing unit 203 to correct the shooting conditions.

[0067] The display processing unit 203 displays a message for re-shooting, as shown in Fig. 11, described below, based on the shooting condition correction instruction information. In this way, the display processing unit 203 displays correction instruction information that prompts the user to immediately correct the lighting conditions and the angle of the camera 116, and performs the display control procedure again. For this reason, it is expected that the re-shot image will not have whiteout, and the shooting determination procedure and known item detection procedure are performed using moving images that are less affected by whiteout, thereby reducing erroneous determination of known images.

[0068] The normalization unit 209 is executed when the known item detection unit 207 is certain to have detected a known image, or when the two-stage highlight detection unit 208 detects that the image has no highlights or has significantly less highlights. The normalization unit 209 extracts an image of a card area from the video captured by the video capture unit 204. The normalization unit 209 then normalizes the number of pixels in both the vertical and horizontal directions of the extracted image area to a size expected as input by the authenticity determination model 202b, thereby normalizing the image to a fixed length. The image area extracted by the normalization unit 209 through cropping is the image area of ​​the card. As a result of the normalization process performed by the normalization unit 209, an image consisting only of the image portion of the card can be input to the authenticity determination model 202b at an appropriate size.

[0069] Further, for the purpose of preventing the top and bottom of the moving image input to the authenticity determination model 202b from being reversed, the normalization unit 209 rotates and resizes the image of the extracted region. The normalization unit 209 performs the rotation and resizing of the image while maintaining the aspect ratio of the original moving image as much as possible. The image size after normalization by the normalization unit 209 can be, for example, an image size of about width m pixels × height n pixels (m < n).

[0070] The authenticity determination unit 21 is executed when the known item detection unit 207 can confirm that a known image has been detected, or when the two-stage white flash detection unit 208 detects that there is no white flash or significantly less white flash in the image. That is, the authenticity determination unit 210 inputs the moving image including the time-series information of the reflected light from the card, which has been normalized by the normalization unit 209, into the authenticity determination model 202b, and obtains the authenticity determination result of the card classified into two binary values of normal (genuine) or abnormal (fake) from the authenticity determination model 202b. The authenticity determination model 202b is based on the temporal change pattern of the reflected light from a predetermined surface of the card medium included in the moving image data of the card medium photographed by the camera 116 in a state where the guide is displayed, and the temporal change pattern of the reflected light from the predetermined surface included in the moving image data obtained by photographing a genuine card medium. And, it executes a determination procedure for classifying the temporal change pattern based on learning. This determination procedure includes a procedure for causing the authenticity determination model 202b, which takes the moving image data as input and outputs the authenticity determination result of the card medium, to classify the temporal change pattern based on learning. Then, by this determination procedure, the authenticity determination model 202b outputs a numerical value for determining the authenticity of the card medium, and this numerical value is classified into two binary values indicating the authenticity determination result of the card.

[0071] The authenticity determination model 202b is characterized by its efficient collection of time-series data from multiple viewpoints or illumination levels through a photography interface that naturally guides the user's actions when photographing a card, and its ability to make maximum use of the temporal change patterns of reflected light that are dependent on the direction of a hologram for both learning and inference. It is also important to note that the data used for authenticity determination is photographed manually. Because the authenticity determination model 202b learns counterfeiting methods generically, without relying on a specific subject, it is a "cold model" that is retrained only when a new counterfeiting method is discovered.

[0072] 4 is a block diagram showing an example of the internal configuration of the learning device 3. In FIG. 4, the master image DB 50 of the learning device 3 is omitted.

[0073] The learning data DB 40 of the learning device 3 stores video data 40a, known item learning data 40b, and authenticity determination learning data 40c. The learning device 3 also includes data extension units 41, 41A, a known item learning unit 42, and an authenticity determination learning unit 43. The server-side learning model DB 45 also stores a known item detection model 45a and an authenticity determination model 45b. The data extension unit 41A and the authenticity determination learning unit 43 included in the learning device 3 are an example of a learning program according to this embodiment.

[0074] The moving image data 40a is, for example, data of moving images captured by a developer on a card medium under various conditions. The moving image data 40a may also include moving images captured by automatically changing the light source, pan, tilt, and focus based on data of known images formed on the card medium. The moving image data 40a includes data of hundreds to tens of thousands of moving images for each known image formed on the card medium.

[0075] The data expansion units 41 and 41A each read video data 40a and expand the video data 40a through various image processing. The data expanded by the data expansion unit 41 is recorded in the learning data DB 40 as known item learning data 40b. The known item learning data 40b is obtained by expanding the video data 40a by the data expansion unit 41 using a conventional procedure. On the other hand, the data expanded by the data expansion unit 41A is recorded in the learning data DB 40 as authenticity determination learning data 40c. The authenticity determination learning data 40c is obtained by expanding the video data 40a by the data expansion unit 41A performing a procedure according to this embodiment.

[0076] Conventionally, when a counterfeit card printed by a printer or a counterfeit image displayed on a display is photographed with a camera, subtle differences from the genuine article can occur due to factors other than the hologram processing, such as differences in the color tone of the printing ink or differences in the brightness of the display. However, it has been difficult to control which features a deep classification model uses to classify and learn.

[0077] Therefore, the data augmentation unit 41A according to the present embodiment executes an "adversarial sampling method (hereinafter referred to as ASM)" that augments positive example data and generates negative example data, which is an adversarial class, from the positive example data. In this embodiment, the data augmentation unit 41A uses ASM to perform a new data augmentation technique, for example, extracting and processing data from a dataset of positive example (genuine) images readily available to manufacturers of marketed products, from which features of the positive examples have been selectively removed, and automatically generating negative example data. Then, in the learning process described below, the authenticity determination learning unit 43 uses the positive example data and the negative example data generated by the data augmentation unit 41A using ASM for learning, and causes the authenticity determination model 45b to learn the presence or absence of target features in an adversarial manner, thereby enabling explicit learning control of the authenticity determination model 45b. The greatest advantage of this technique is that it can match the positive example data and the negative example data as much as possible, except for features to be contrasted. This allows the authenticity determination model 45b to perform learning for authenticity determination by focusing only on the artificially intended features.

[0078] The data extension unit 41A includes an extraction unit 411, a duplication unit 412, and an extension unit 413. The following describes the process performed by the data extension unit 41A to generate the authenticity determination learning data 40c.

[0079] The extraction unit 411 extracts any number of video data 40a from multiple video data 40a captured in chronological order on a card medium, and executes an extraction procedure to extract, for each extracted video data 40a, the arrangement of images contained in the video data 40a as an initial image arrangement.

[0080] The duplication unit 412 executes a duplication procedure to generate an array of static images by duplicating an image selected from the initial image array by an arbitrary number of times.

[0081] The expansion unit 413 copies the array of static images, performs a predetermined expansion process on any static images included in the array of copied static images, generates an array of dynamic images, and executes an expansion procedure in which the array of static images and the array of dynamic images are recorded in the learning data DB 40 as authenticity determination learning data 40c for learning the card medium.

[0082] The extension unit 413 generates an array of dynamic images by an extension process that combines independent image processing, which performs arbitrary image processing on one or more static images extracted from the array of static images, and continuous image processing, which performs arbitrary image processing on multiple static images extracted from the array of static images. The arbitrary image processing includes at least one of adding blur to a static image, cutting out a rectangular area from the static image, enlarging, reducing, rotating, translating the static image, changing the brightness of a part of the static image, and cutting out the edges of the static image.

[0083] Known item learning data 40b is data that is read by known item learning unit 42 to learn known items into known item detection model 45a. Known item learning unit 42 reads known item detection model 45a from server-side learning model DB 45, inputs known item learning data 40b, and causes known item detection model 45a to learn known items.

[0084] The authenticity determination learning data 40c is data that the authenticity determination learning unit 43 reads in to train the authenticity determination model 45b to perform authenticity determination. The authenticity determination learning unit 43 reads the authenticity determination model 45b from the server-side learning model DB 45, inputs the authenticity determination learning data 40c, and trains the authenticity determination model 45b to perform authenticity determination. The authenticity determination learning unit 43 according to this embodiment trains the authenticity determination model 45b, into which the initial image array, static image array, and dynamic image array are loaded as the authenticity determination learning data 40c, on the card medium, and executes an authenticity determination procedure in which the trained authenticity determination model 45b is recorded in the server-side learning model DB 45 (an example of a learning model recording unit). The authenticity determination learning unit 43 then saves the trained authenticity determination model 45b in the server-side learning model DB 45.

[0085] <Learning process> Next, the learning process will be described with reference to FIGS. FIG. 5 is a flowchart showing an example of a known item learning process.

[0086] First, the data extension unit 41 acquires video data 40a as an image (original data) to be detected from the learning data DB 40 (S1). The image (original data) to be detected includes a video of a known image formed on a card medium.

[0087] Next, the data expansion unit 41 expands the number of data by expanding the image (original data) of the detection target acquired from the video data 40a using a data expansion algorithm (S2), and stores the expanded data as known item learning data 40b in the learning data DB 40. The data expansion algorithm is used by the data expansion unit 41 to perform image processing such as adding blur to the known image, cutting out a rectangular area from the known image, enlarging, reducing, rotating, and translating the known image, changing the brightness of a part of the known image, and cutting out the edges of the known image.

[0088] Next, the data extension unit 41 inputs the known item learning data 40b into the known item detection model 45a read from the server-side learning model DB 45, thereby learning known items based on the original data with the expanded number of data (S3).The data extension unit 41 then saves the learned known item detection model 45a in the server-side learning model DB 45 (S4), and ends this process.

[0089] FIG. 6 is a flowchart showing an example of the authenticity determination learning process.

[0090] First, the data extension unit 41 uses a data extension algorithm to extend the image (original data) of the detection target obtained from the video data 40a to expand the number of pieces of data (S11), and stores the expanded data as authenticity determination learning data 40c in the learning data DB 40. The processing in step S11 is called the learning data extension processing, and will be described in detail later with reference to Figures 7 and 8. Note that the learning data extension processing may be performed before the authenticity determination learning processing as a processing independent of the authenticity determination learning processing.

[0091] Next, the authenticity determination learning unit 43 acquires the authenticity determination learning data 40c from the learning data DB 40 (S12). Next, the authenticity determination learning unit 43 inputs the authenticity determination learning data 40c into the authenticity determination model 45b read from the server-side learning model DB 45, thereby training the model 45b to determine the authenticity of a known image (S13).

[0092] Then, the authenticity determination learning unit 43 stores the learned authenticity determination model 45b in the server-side learning model DB 45 (S14), and ends this process.

[0093] <Specific example of training data augmentation processing> Here, a specific example of the training data extension process shown in step S11 will be described.

[0094] 7 is a flowchart showing an example of the training data extension process, which is performed by the functional units of the data extension unit 41A shown in FIG. FIG. 8 shows examples of images from the video data 40a to the generation of the authenticity determination learning data 40c.

[0095] Before starting the training data augmentation process, the developer checks the latest status of the metafile and initializes the workspace of the training data DB 40 (S20). Multiple video data 40a used as positive example data are stored in a predetermined synchronized directory of the training data DB 40. Therefore, the developer checks that the metafile listing the multiple video data 40a is up to date. The developer also initializes two workspaces that store images generated by the static ASM and dynamic ASM (described below) so that they are empty and do not retain any past artifacts. The areas for both workspaces are both provided in the training data DB 40. In step S20, the developer prepares the environment used in the training data augmentation process, thereby preventing data generated in previous processes from being mixed into subsequent processes and contaminating the training data.

[0096] As shown in FIG. 8, for example, a developer has prepared a large number of video data 40a in the learning data DB 40, each of which is a collection of images of a genuine card medium. The video data 40a is a file encoded in a video format such as mp4. The developer has captured images of a physical card medium as a time-series video under various conditions, for example, by changing the angle and light intensity of the light source, or by changing the orientation and size. The video data 40a includes images of a genuine card medium that has been embossed and hologram-processed, captured using a camera application built into a smartphone or the like.

[0097] (1) in FIG. 8 is an example of a population set N of positive example data. A developer has captured video data 40a of all types of physical card media, and this set is taken as the population set N of positive example data. The ASM according to this embodiment extends the population set N of positive example data. However, since applying ASM to all positive example data would result in an excessive increase in the amount of data, ASM is applied in a way that maintains sufficient statistical diversity while ensuring cost-effectiveness and class balance.

[0098] Here, the total number of video data 40a belonging to the population set N of positive example data is defined as n. If the i-th video data 40a among n is expressed as vi, then N = (v1, v2, ..., vi, ... vn). Each of the video data 40a is video data of 32 frames and is RGB color data.

[0099] When the training data expansion process begins, the extraction unit 411 randomly extracts L pieces of video data 40a as samples from the population N of positive example data (S21). Therefore, vi is calculated for each of the extracted L pieces of video data 40a. The developer sets the L pieces of extracted data in advance so that the required number of negative example data is met by combining both static ASM and dynamic ASM sequences, which will be described later. The developer then has the extraction unit 411 execute the extraction procedure of step S21, thereby adjusting the trade-off between sample size and class balance and preventing excessive capacity due to subsequent data amplification.

[0100] The following describes the process, focusing on one piece of video data vi out of the L pieces of extracted video data 40a. Although not shown, the same process as for the video data vi is also performed on other video data 40a other than the video data vi.

[0101] Next, the extraction unit 411 generates an initial image array based on the video data vi extracted from the population N of positive example data (S22). The initial image array shown in (2) of FIG. 8 is 32 bitmap arrays a=[f1, f2, ... f32] obtained by the extraction unit 411 decoding the video data vi extracted from the total number n of video data 40a and capturing any frames of the decoded video data vi. Here, fx is the xth frame image, a bitmap image with fixed width and height, encoded in any format such as PNG (Portable Network Graphics) or BMP (Bitmap Image). The number 32 is an arbitrary number and may be any other number.

[0102] The initial image array a shown in (2) of Figure 8 represents the state of a subject captured by moving the camera. For example, the subject is captured on the left side in frame 1, but is captured on the right side in frame 32. In this way, the video data vi includes 32 frames of images captured continuously in a time series.

[0103] Next, the duplication unit 412 duplicates one frame image randomly selected from the initial image array to generate a static image array (S23). This process is performed to create negative example data that has the same properties as the positive example data but has removed temporal changes from one image frame. By randomly extracting only one frame from the time axis of the video, the duplication unit 412 can remove temporal information while keeping the spatial information of the image fixed. In other words, in subsequent processing, a static image array can be obtained as a base frame that clarifies the conflict axis of, for example, "missing features due to hologram processing."

[0104] The static image array b shown in Figure 8 (3) is an array of bitmap images created by the duplication unit 412 randomly selecting one image from the initial image array and arranging 32 copies of that image. In other words, an array of 32 identical images is a static image array. For this reason, the static image array has the property that while it perfectly matches the positive example data spatially, pixel differences in the time series direction are completely eliminated. This is a "static negative example in which the diffraction pattern specific to a hologram does not appear," and it serves to make the presence or absence of a hologram itself the sole basis for identification.

[0105] In step S23, it is assumed that the second frame image of the initial image array a is randomly selected. For example, the second bitmap image f2 is randomly selected from the initial image array a=[f1, f2, ... f32].

[0106] Next, the expansion unit 413 replicates the entire static image array (S24). For example, the static image array is expressed as a bitmap array b=[f1, f2, ... f32] obtained by replicating the ith bitmap fi randomly selected from the initial image array a=[f1, f2, ... f32]. At this time, f1 to f32 of bitmap b all hold the same data as fi of a. The process of replicating the ith bitmap fi randomly selected from the initial image array a shown in step S24 is called "static ASM." The static image array b is data that does not include pixel changes between frames caused by the diffractive optical structure of the hologram processing.

[0107] Next, the expansion unit 413 controls the amount of static ASMs and the amount of dynamic ASMs (described later) generated using a predetermined image generation ratio (S25). This process is performed to adjust the configuration of the authenticity determination training data 40c used for authenticity determination training. Simply having the authenticity determination model 45b learn from static image arrays as negative example data carries the risk that the authenticity determination model 45b will simply learn from the negative example data based on the fact that there is no time-series change at the pixel level, and will make an authenticity determination based on whether the pixel-by-pixel difference between images for each frame is zero.

[0108] Therefore, the extension unit 413 adjusts the number of static image arrays and the number of dynamic image arrays (described later) at a predetermined image generation ratio. For example, an image generation ratio (e.g., 1:2) is used that makes the amount of static ASMs generated less than the amount of dynamic ASMs generated. However, an image generation ratio (e.g., 1:1) that makes the amount of static ASMs generated the same as the amount of dynamic ASMs generated may also be used, or an image generation ratio (e.g., 2:1) that makes the amount of static ASMs generated greater than the amount of dynamic ASMs generated may also be used. The extension unit 413 adds static image arrays corresponding to the amount of static ASMs generated determined by the image generation ratio to the negative example data of the authenticity determination training data 40c in the training data DB 40.

[0109] Furthermore, the expansion unit 413 performs independent and continuous image transformations, which are explained below, according to the generation amount of the dynamic ASM, which is determined by the image generation ratio. By using these image transformations, the expansion unit 413 can generate an array of dynamic images, which are pseudo-moving images that do not include time-series features resulting from hologram processing, but only have time-series differences.

[0110] For example, the expansion unit 413 performs independent image transformation (an example of independent image processing) on ​​the static image array b (S26). Independent image transformation is a procedure in which the expansion unit 413 performs geometric transformation processing on static images randomly extracted from the static image array b. In other words, the expansion unit 413 performs data expansion on each static image included in the static image array b using independent random parameters (e.g., probability values). Other independent image transformations include, for example, adding blur to a known image, cropping a rectangular area from the known image, enlarging, reducing, rotating, and translating the known image, changing the brightness of a portion of the known image (e.g., adding blown-out highlights), and cropping the edges of the known image. Frame 1 shown in (4) of Figure 8 is an image in which blurring and rotation have been added to the subject. Frame 2 is an image in which the subject has been enlarged and horizontally translated. Frame N is an image in which the subject has been reduced in size and translated horizontally and vertically.

[0111] Next, the expansion unit 413 performs continuous image transformation on the static image array b (S27). Continuous image transformation is a process in which the expansion unit 413 performs data expansion by smoothly varying the rotation angle or brightness of the static image linearly or sinusoidally in the time series direction using parameters that gradually change in accordance with the frame numbers of the static image array b. Continuous image transformation generates a pseudo-continuous image that involves changes in pixels but does not involve changes in hologram characteristics.

[0112] The dynamic image array shown in (5) of FIG. 8 is a bitmap image array obtained by applying continuous image transformation to all or part of the static image array b by the expansion unit 413. For example, the dynamic image array is represented as an array c=[f1, f2, ... f32] obtained by transforming and mapping all elements fi of the static image array b=[f1, f2, ... f32]. The expansion unit 413 generates a dynamic image array c=[f1, f2, ... f32] obtained by transforming and mapping all elements fi of the static image array b=[f1, f2, ... f32]. The process of replicating and transforming the static image array shown in steps S26 and S27 is called "dynamic ASM." (5) of FIG. 8 shows an example of an image that has undergone continuous image transformation, in which the image angle is gradually rotated clockwise.

[0113] The expansion unit 413 adds the dynamic image array c generated by the dynamic ASM in steps S26 and S27 to the negative example data. The dynamic image array c generated by the expansion unit 413 by combining independent image transformation and continuous image transformation with respect to the static image array b reproduces the natural fluctuations that occur when a person photographs a card medium from different angles, for example. This allows positive example data and negative example data to be distinguished based on the "unique characteristics of the hologram" rather than the "amount of pixel change."

[0114] The expansion unit 413 then determines whether the total number of positive example data and the total number of negative example data are balanced within the allowable error ε (S28). The allowable error ε may be any number. The processing in step S28 is performed to prevent learning bias due to an imbalance in the number of positive example data and negative example data.

[0115] If the extension unit 413 determines that the data are not balanced within the allowable error ε (NO in S28), the process returns to step S21, where the extraction procedure is executed and the authenticity determination training data 40c is recreated. In this recreate process, processes such as resampling and thinning of images from the array are automatically executed, and the authenticity determination training data 40c is corrected so that it is not biased due to class imbalance.

[0116] On the other hand, if the expansion unit 413 determines that the balance is within the allowable error ε (YES in S28), it stores the generated data (initial image array a, static image array b, and dynamic image array c) in the training data DB 40 as authenticity determination training data 40c (S29), and ends this process. This process generates authenticity determination training data 40c in which two types of negative example data (static image array b and dynamic image array c) coexist at a predetermined ratio (image generation ratio). That is, passing the determination in step S28 results in the authenticity determination training data 40c being a conflicting data set based solely on the presence or absence of a hologram, for example. Therefore, the authenticity determination training data 40c is safely passed to the subsequent learning process by the authenticity determination training unit 43.

[0117] <Guide frame display processing> 9 and 10, a description will be given of a guide frame display process performed by the display processing unit 203 of the information processing terminal 2. FIG. 9 is a diagram showing an example of a guide frame display based on the guide frame display process performed by the display processing unit 203.

[0118] The left side of Fig. 9 shows a state in which the frame of the card Cd is displayed within a guide frame Fm displayed on the screen of the display device 119 by the display processing unit 203. The guide frame Fm is an example of an image that specifies the display position of the card medium. When this state occurs, that is, when it is detected that the frame of the card Cd has fit within the guide frame Fm, the display processing unit 203 displays a sign such as "OK" on the screen.

[0119] Next, the display processing unit 203 provides guidance to change the position of the guide frame displayed on the screen to a position different from the previous displayed position. As shown on the right side of Fig. 9, the display processing unit 203 shifts the display position of the guide frame Fm on the screen diagonally upward to the right. Upon seeing the guide displayed with the display position shifted, the user moves the information processing terminal 2 that is being held in order to fit the frame of the card Cd into the card frame Fm.

[0120] By performing such processing by the display processing unit 203, the position of the information processing terminal 2 held by the user changes up, down, left, right, etc. in accordance with the change in the display position of the guide frame. In other words, the angle at which the card is photographed by the camera 116 of the information processing terminal 2 changes. Note that the number of times the display position of the guide frame is changed by the display processing unit 203 may be any number of times as long as it is at least once or more.

[0121] 10 is a diagram showing an example of changes in the photographing angle of a card taken by camera 116. When the display processing unit 203 changes the display position of the guide frame, the photographing angle of the card taken by camera 116 changes to various angles, as shown in FIG.

[0122] Then, video shooting unit 204 records, as a video, the images captured by camera 116 with the imaging angle changed. By recording, as a video, the images captured by camera 116 with the imaging angle changed, it becomes possible to include, in the video shot by video shooting unit 204, a feature that essentially has time axis information, that is, a temporal change pattern of reflected light according to the imaging angle.

[0123] The authenticity determination system 100 according to the present embodiment determines the authenticity of cards that have undergone fine hologram and embossing processes. Embossing is, for example, a process of printing a transparent varnish on the surface (an example of the first surface) of a card to form a concave and convex portion. Therefore, when such a card is photographed, the temporal change pattern of the reflected light from the card changes in a subtle and diverse manner as the photographing angle of the camera 116 changes. In other words, the authenticity determination model 202b according to the present embodiment learns the subtle and diverse changes in the temporal change pattern of the reflected light from such a card. The surface treatment of the card is not limited to both fine hologram and embossing, and may be either one of them. Furthermore, the surface treatment of the card may be other than hologram or embossing, such as gold foil stamping, silver foil stamping, or mother-of-pearl inlay, as long as it can change the reflection state of the reflected light from the card surface.

[0124] <Example of an image with overexposed highlights> Figure 11 shows an example of an image that contains overexposed highlights. When a card that has been treated to make it more reflective is photographed under bright lighting, overexposed highlights may appear in the image.

[0125] For example, a blown-out highlight Wh occurs in the area indicated by the dashed line in FIG. 11. Because the blown-out highlight Wh covers the photographed card Cd, part of the frame of the card Cd and part of the character image formed on the card Cd appear white. Note that the card frame Fm is displayed by the display processing unit 203 and is therefore not affected by the blown-out highlight Wh. When blown-out highlights Wh cover the card Cd in this way, information necessary for determining the authenticity of the card Cd may be missing. For this reason, the display processing unit 203 displays a message saying, "The screen is too bright. Please darken the surroundings and take the photograph again," instructing the user to take the photograph again. When the user sees this message, it is expected that they will photograph the card Cd again.

[0126] <Authenticity determination process using an authenticity determination system> Next, the authentication determination process by the authentication determination system 100 according to this embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the procedure of the authentication determination process by the authentication determination system 100.

[0127] First, the display processing unit 203 (see FIG. 3) of the information processing terminal 2 executes a guide frame display process (S31). The guide frame display process of step S31 starts when the user starts an application incorporating a program that executes the authenticity determination process according to this embodiment and selects a menu for determining the authenticity of a card. The contents of the guide frame display process performed in step S31 will be described in detail with reference to FIG. 13 described below.

[0128] Next, the camera 116 starts taking pictures and recording the image data taken by the camera 116 as a moving image (S32). The process of step S32 continues while the guide frame is displayed on the screen of the display device 119 in step S31 and the user moves the information processing terminal 2 up, down, left, right, etc. to fit the frame of the card into the guide frame.

[0129] Next, the card area detection unit 205 extracts a card area from the image data captured by the camera 116 in step S32 (S33). Next, the photography determination unit 206 performs photography determination processing (S34). The photography determination processing will be described in detail later with reference to FIG. 14.

[0130] Next, the known item detection unit 207 performs a known item detection process (S35). The known item detection process will be described in detail with reference to FIG. 15 below. Next, the known item detection unit 207 determines whether the certainty score s obtained by the known item detection process falls within a range of predetermined thresholds t1 to t2 (S36). If the certainty score s exceeds the threshold t2 (NO in S36), it is sufficiently certain that a known image has been detected as a known item, and the process proceeds to step S38.

[0131] On the other hand, if the certainty score s falls within the range of the predetermined thresholds t1 to t2 (YES in S36), it is not certain that a known image has been detected as a known item, and the process proceeds to step S37. Then, the two-stage blown out highlight detection unit 208 performs two-stage blown out highlight detection processing (S37). The two-stage blown out highlight detection processing will be described in detail later with reference to FIG. 16.

[0132] If the certainty score s is less than the threshold t1, it is highly likely that the subject is outside the guide frame Fm or is obscured. If the certainty score s is equal to or greater than the threshold t2, the authenticity of the image can be determined with sufficient confidence, and the two-stage blown-out highlight detection process in step S37 is unnecessary.

[0133] Furthermore, in this embodiment, thresholds t1 to t2 are set, but t1 may be set to 0. In this case, if the threshold value t2 is less than the threshold value t2, it is not certain that a known image has been detected, and the two-stage blown-out highlight detection process of step S37 may be performed.

[0134] If it is sufficiently certain that a known image has been detected in step S36, or after the two-stage blown-out highlight detection process in step S37, the normalization unit 209 normalizes the size of the video recorded in step S32 to a size suitable for input to the authenticity determination model 202b (S38). Next, the authenticity determination unit 210 performs an authenticity determination process (S39). The authenticity determination process performed in step S39 will be described in detail with reference to FIG. 17 below.

[0135] (Guide frame display processing) Next, the guide frame display process performed in step S31 in Fig. 12 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the procedure for the guide frame display process performed by the display processing unit 203.

[0136] First, the display processing unit 203 displays a guide frame on the screen of the display device 119 (S41). Next, the display processing unit 203 determines whether the frame of the card included in the image data captured by the camera 116 in step S32 fits within the guide frame (S42). If it is determined that the card does not fit within the guide frame (NO in S42), the display processing unit 203 continues to make the determination in step S42.

[0137] On the other hand, if it is determined that the card has fit into the guide frame (YES in S42), the display processing unit 203 displays a sign such as "OK" on the screen to indicate that the card has fit into the guide frame (S43). Note that the sign indicating that the card has fit into the guide frame is not limited to the letters "OK" and may be other letters. Furthermore, the sign may not be letters, but may be other forms of notification to the user, such as a change in the color or shape of the guide frame, or the emission of a sound.

[0138] Next, the display processing unit 203 determines whether or not the termination condition of the shooting determination process by the shooting determination unit 206 (see FIG. 3) is satisfied (S44). The termination condition of the shooting determination process is that both the determination result of "success" or "failure" by the shooting determination process and the captured image (moving image) by the camera 116 in the case of a "success" determination are obtained.

[0139] A moving image determined to be "successful" is obtained when the shooting determination process for all guide frames has been completed. If it is determined in step S44 that the conditions for terminating the shooting determination process have not been met (NO in S44), the display processing unit 203 changes the display position of the guide frames on the screen (S45). After processing step S45, the display processing unit 203 returns to step S41 and performs processing.

[0140] On the other hand, if it is determined in step S44 that the termination condition for the shooting determination process is satisfied (YES in S44), the guide frame display process is terminated by the display processing unit 203. Note that the guide frame display process by the display processing unit 203 may be terminated at a predetermined timing after the determination result of the shooting determination process is output.

[0141] (Shooting judgment processing) Next, the shooting determination process performed in step S34 of Fig. 12 will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the procedure for the shooting determination process performed by the shooting determination unit 206.

[0142] First, the photography determination unit 206 extracts the position of the card from the card area detected in the photographed image in step S33 of Fig. 12 (S51). Next, the photography determination unit 206 determines whether the movement vector of the card position extracted in step S51 and the movement vector of the six-axis sensor 117 of the information processing terminal 2 are moving simultaneously in the same direction (S52).

[0143] If it is determined that the movement vector of the card position and the movement vector of the six-axis sensor 117 are moving simultaneously in the same direction (YES in S52), the photography determination unit 206 determines that the camera 116 is photographing a physical card medium (S53). On the other hand, if it is determined that the movement vector of the card position and the movement vector of the six-axis sensor 117 are not moving simultaneously in the same direction (NO in S52), the photography determination unit 206 determines that the camera 116 is photographing a subject other than a physical card medium (S54). After processing step S54 or step S55, the photography determination process by the photography determination unit 206 ends.

[0144] The determination in step S52 can be made by analyzing the movement directions of both movement vectors, but can also be made based on the magnitude of the correlation coefficient between both movement vectors, etc.

[0145] (Known item detection process) Next, the known item detection process performed in step S35 of Fig. 12 will be described with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the procedure of the known item detection process performed by the known item detection unit 207.

[0146] First, the known item detection unit 207 reads an image captured by the video image capturing unit 204, and inputs the image to the known item detection model 202a read from the local-side inference model DB 202 (S61).

[0147] The known item detection model 202a estimates the presence or absence of an identifier of the card Cd and a written language (Japanese, Chinese, English, etc.) based on the input image (S62). Then, the known item detection model 202a outputs a certainty score s (S63). After the processing of step S63, the known item detection processing by the known item detection unit 207 ends.

[0148] (Two-stage overexposure detection processing) Next, the two-stage blown-out highlight detection process performed in step S37 of Fig. 12 will be described with reference to Fig. 16. Fig. 16 is a flowchart showing an example of the procedure for the two-stage blown-out highlight detection process performed by the two-stage blown-out highlight detection unit 208.

[0149] First, the two-stage blown-out highlight detection unit 208 performs a global analysis in steps S71 to S73. In the global analysis, the two-stage blown-out highlight detection unit 208 converts an image captured by the video capture unit 204 and input (referred to as an input image) into a luminance image (grayscale image) (S71). The input image is, for example, an 8-bit, 3-channel video image. After formal validation of the input image (changing the image size, etc.), the two-stage blown-out highlight detection unit 208 converts the input image into a grayscale image, i.e., a 1-channel luminance image, if the input image is normal. This is because blown-out highlight determination uses only luminance values.

[0150] Next, in a first analysis step, the two-stage blown-out highlight detection unit 208 analyzes the percentage of pixels in the image of the card medium whose brightness values ​​exceed a predetermined brightness threshold. The first analysis step involves converting the entire image of the card medium into a brightness image and comparing the brightness value of each pixel in the brightness image with the brightness threshold. Specifically, the two-stage blown-out highlight detection unit 208 calculates the percentage of pixels in the entire converted brightness image whose brightness values ​​exceed a certain brightness threshold (e.g., a brightness value of 250) (S72). Here, the two-stage blown-out highlight detection unit 208 calculates the frequency of brightness values ​​for each pixel as a histogram. The number of brightness value gradations is 256 bins ranging from 0 to 255. Each bin represents a specific brightness value. The two-stage blown-out highlight detection unit 208 then calculates the number of pixels in the entire image and calculates the total number of bins in the histogram whose brightness values ​​exceed the brightness threshold, i.e., whose brightness values ​​are greater than or equal to 250 and less than or equal to 255.

[0151] Next, the two-stage blown-out highlight detection unit 208 calculates the ratio of the total number of bins exceeding the brightness threshold to the total number of pixels in the entire image. This calculation is performed to determine the ratio of high-brightness pixels that can be considered "almost white" to the entire image. The two-stage blown-out highlight detection unit 208 then determines whether the calculated ratio exceeds a preset ratio threshold (S73). This ratio threshold is preferably less than 10%, for example.

[0152] If it is determined that the proportion of pixels exceeding the brightness threshold exceeds a predetermined proportion threshold (YES in S73), it is considered that blown-out highlights have occurred over a wide area. Therefore, the two-stage blown-out highlight detection unit 208 outputs shooting condition correction instruction information to the display processing unit 203 (S79). Furthermore, if the proportion of pixels exceeds a predetermined pixel proportion threshold, the two-stage blown-out highlight detection unit 208 performs a first warning procedure to warn of the possibility of a poorly shot area.

[0153] If it is determined that the proportion of pixels exceeding the brightness value threshold does not exceed the preset proportion threshold (is equal to or less than the proportion threshold) (NO in S73), the two-stage blown-out highlight detection unit 208 performs local analysis shown in steps S74 to S78.

[0154] In the local analysis, if the two-stage blown-out highlight detection unit 208 determines in the first analysis step that the pixel ratio does not exceed the pixel ratio threshold, it divides the image of the card medium into blocks of a predetermined size and analyzes the relationship between the high-luminance density and the predetermined high-luminance density threshold for each block. Specifically, the two-stage blown-out highlight detection unit 208 first divides the luminance image into small blocks (a 4x4 grid) (S74). Here, the luminance image is divided into 16 blocks by dividing the original image size into 4 vertically and 4 horizontally, and the two-stage blown-out highlight detection unit 208 calculates the height and width of each block in terms of the number of pixels. Note that the number of blocks (16) into which the two-stage blown-out highlight detection unit 208 divides the luminance image is merely an example; the original image size may be divided into 2 vertically and 2 horizontally, resulting in 4 blocks, or the original image size may be divided into 5 vertically and 5 horizontally, resulting in 25 blocks.

[0155] Next, the two-stage blown-out highlight detection unit 208 performs a loop process for each divided block (S75). In this loop process, the two-stage blown-out highlight detection unit 208 first scans one of the blocks and counts the number of pixels bC whose luminance value exceeds 250, for example (S76).

[0156] Next, the two-stage blown-out highlight detection unit 208 calculates the high luminance density bR of the block (S77). In the second analysis procedure, the high luminance density bR is calculated as the ratio of the number of pixels whose luminance values ​​are equal to or greater than the luminance threshold value to the total number of pixels in the block. Specifically, it is calculated using the following formula (1): High luminance density bR = number of pixels bC / total number of pixels in the block × 100 …(1)

[0157] Next, the two-stage blown-out highlight detection unit 208 determines whether the high luminance density bR is equal to or greater than a preset density threshold (S78). The density threshold may be set to, for example, 10% or less. By calculating the high luminance density bR, the two-stage blown-out highlight detection unit 208 detects local overexposure that causes blown-out highlights in the luminance image.

[0158] On the other hand, if the two-stage blown-out highlight detection unit 208 determines that the high luminance density bR is not equal to or greater than the preset density threshold (NO in S78), it performs local analysis on the next block. For example, after processing the block in the upper left of the original image, it processes the blocks to the right, and after processing the rightmost block, it processes the block one block below and to the left. This loop process is then performed on all blocks.

[0159] If the two-stage blown-out highlight detection unit 208 makes a NO determination in step S78 as a result of performing local analysis on all blocks, the possibility of blown-out highlights is low, and the process ends.

[0160] On the other hand, if the two-stage blown-out highlight detection unit 208 determines that the high luminance density bR is equal to or greater than the preset density threshold (YES in S78), it exits the loop even if local analysis has not been performed on all blocks. In other words, the two-stage blown-out highlight detection unit 208 exits the loop if it determines that the high luminance density bR of even one block is equal to or greater than the preset density threshold.

[0161] If the two-stage blown-out highlight detection unit 208 determines through global analysis that the pixel ratio exceeds the ratio threshold (YES in S73), or if the local analysis determines that the high luminance density bR is equal to or greater than a preset density threshold (YES in S78), it outputs shooting condition correction instruction information to the display processing unit 203 (S79). In this way, if there is a block where the high luminance density is equal to or greater than the high luminance density threshold, the two-stage blown-out highlight detection unit 208 performs a second warning procedure that warns of the possibility of a poorly shot area.

[0162] When the display processing unit 203 receives the correction instruction information for the shooting conditions, it displays a warning of the possibility of overexposure as shown in Fig. 11 (S80), and returns the process to step S33 in Fig. 12 (detecting the card area from the captured image). In this way, even in a state affected by the automatic exposure control of the camera 116, the display processing unit 203 can detect only the light that impairs the information of the subject and display a warning of the possibility of overexposure. Thereafter, the process from step S33 onwards is performed again.

[0163] In this way, the two-stage blown-out highlight detection unit 208 determines the final result as true, i.e., blown-out highlights detected, if either the global analysis or the local analysis is true. If both the global analysis and the local analysis are false, blown-out highlights are not detected.

[0164] (Authenticity determination process) Next, the authenticity determination process performed in step S39 of Fig. 12 will be described with reference to Fig. 17. Fig. 17 is a flowchart showing an example of the procedure for the authenticity determination process performed by the authenticity determination unit 210.

[0165] First, the authenticity determination unit 210 inputs the moving image of the card area normalized in step S38 of Fig. 12 to the authenticity determination model 202b (S81). Next, the authenticity determination model 202b analyzes the pattern of change over time of reflected light obtained from the image of the card area that was captured by the camera 116 and normalized, and classifies the pattern of change over time on a learning basis (S82). As described above, the authenticity determination model 202b can determine the authenticity of the card medium itself, even for an image that shows reflected light that changes over time, without taking into account the influence of that reflected light.

[0166] Next, the authenticity determining unit 210 outputs the result of the verification by the authenticity determining model 202b as the authenticity determination result of the card (S83).

[0167] The authenticity determination model 202b analyzes the temporal change pattern of the reflected light obtained from the image of the card area, and based on the results of learning-based classification of the temporal change pattern, obtains an authenticity determination result that the card photographed by the camera 116 is a regular (genuine) card, and outputs this authenticity determination result.

[0168] On the other hand, the authenticity determination model 202b analyzes the pattern of change over time of the reflected light obtained from the image of the card area, and based on the result of classifying the pattern of change over time based on learning, obtains an authenticity determination result that the card photographed by the camera 116 is an illegal (counterfeit) card, and outputs this authenticity determination result. After the processing of step S83, the authenticity determination processing by the authenticity determination unit 210 ends.

[0169] In the first embodiment described above, the initial image array a generated from the video data 40a is the subject of machine learning as positive example data, and the static image array b and the dynamic image array c after image conversion generated by the expansion unit 413 are the subjects of machine learning as negative example data. The authenticity determination model 45b clearly contrasts feature differences based on the presence or absence of hologram processing, embossing, etc. applied to the card medium between two classes, positive and negative, making it possible to effectively learn features important for classification.

[0170] The data expansion unit 41A uses ASM to artificially select subtle features that serve as classification criteria, generating opposing authenticity determination training data 40c in which positive and negative example data are distinguished solely by the presence or absence of features possessed by genuine (authentic) card media. ASM is used to generate negative example data by intentionally removing time-series optical changes specific to hologram processing from the positive example data. Furthermore, unlike negative example data of counterfeit cards obtained by photographing images printed by a printer or displayed on a display, the negative example data generated by static ASM and dynamic ASM utilizes positive example data immediately before input into the authenticity determination model 45b. Therefore, the negative example data generated by static ASM and dynamic ASM have properties similar to those of genuine (authentic) card media images, except for the features artificially added by dynamic ASM. Furthermore, the negative example data generated by the static ASM and dynamic ASM maintains the same characteristics as the positive example data on a frame-by-frame basis, but by contrasting only the characteristics essential for classification between the positive example data and the negative example data, the authenticity determination model 45b can learn these characteristics as the basis for classification.

[0171] As described above, the data expansion unit 41A artificially generates a large amount of negative example data from only positive example data, significantly reducing the cost of developers collecting large quantities of negative example data, which is difficult to collect in practice. Furthermore, obtaining a large amount of negative example data is extremely advantageous for developers, especially in situations where it is difficult to collect data on counterfeit card media during the card media manufacturing process or market distribution stage. Furthermore, because the data expansion unit 41A automatically generates negative example data using only images of positive example data as the source material, it reduces the labor costs required to generate negative example data when operating large-scale card media, enabling rapid and widespread authentication of even new counterfeit card media. For example, the system has high scalability from an engineering perspective, as it can easily adapt to expansions in card media production lines and market size.

[0172] Additionally, the data expansion unit 41A artificially generates negative example data by specializing in the presence or absence of the characteristic diffractive optical structure of hologram-processed card media. This allows the authentication determination model 45b to explicitly learn only the essential features of card media that are essential for authenticity determination. This explicit feature selection prevents the authentication determination model 45b (authenticity determination model 202b) used in the information processing terminal 2, such as a smartphone, from relying on secondary and accidental features (print quality, noise, moiré patterns, etc.) to determine the authenticity of card media, thereby achieving robust and highly accurate authentication determination. In particular, the authentication determination model 45b (authenticity determination model 202b) is expected to maintain high accuracy in authenticity determination even for images of card media captured in unknown or changed environments (such as changes in lighting conditions, diverse imaging devices, and angles).

[0173] The data expansion process in the data expansion unit 41A is not limited to image data of a card medium. For example, it can also be applied in principle to time-series data acquired in different fields, such as audio, vibration waveforms, and electromagnetic signals. Therefore, the data expansion process according to this embodiment and the authenticity determination learning process are expected to be applied to anomaly detection and authenticity determination in a wide range of engineering fields.

[0174] Furthermore, when a known subject is present within the frame and the certainty score of the subject detected by the known item detection unit 207 is less than the certainty threshold, a two-stage blown-out highlight detection process, consisting of a global analysis and a local analysis, is performed on the captured image. The two-stage blown-out highlight detection process analyzes the brightness characteristics of the entire image and each block to detect loss of information about the known subject due to blown-out highlights. The authenticity determination unit 210 of the information processing terminal 2 analyzes the time-series information of the reflected light from the card obtained by the camera 116, i.e., the temporal change pattern of the reflected light, and classifies the temporal change pattern based on learning to determine the authenticity of the card. Therefore, according to this embodiment, the authenticity of a card can be determined without adding or connecting an authenticity determination device, such as a wide-angle microscope, to the information processing terminal 2.

[0175] Furthermore, in the first embodiment, a simple threshold-based image processing method is used to detect blown-out highlights across the entire image and for each block, while maintaining high computational efficiency that enables real-time processing on a smartphone. To ensure versatility that is not dependent on specific device-specific functions, the two-stage blown-out highlight detection unit 208 detects blown-out highlights only through image analysis of image data after AE (Auto Exposure) or AI correction, rather than raw data specific to the camera device.

[0176] Furthermore, unlike intentional obscuration (illegal manipulation) caused by covering the subject with a finger or a card other than the subject, blown-out highlights can frequently occur when a well-intentioned user takes a photo due to lighting conditions or shooting angle. Therefore, when the two-stage blown-out highlight detection unit 208 detects blown-out highlights, the display processing unit 203 immediately prompts the user to correct the shooting conditions, such as by dimming the lighting or adjusting the camera angle. This reduces unnecessary reshoots and false detections of blown-out highlights, and enables blown-out highlight detection to be achieved with the minimum number of activations of each AI in the information processing terminal 2.

[0177] The two-stage blown-out highlight detection unit 208 according to the first embodiment detects blown-out highlights in an image in two stages: global analysis and local analysis. Therefore, the two-stage blown-out highlight detection unit 208 can detect blown-out highlights that occur in a captured image even if excessive light is not incident on the image sensor or the like.

[0178] Furthermore, the two-stage blown-out highlight detection unit 208 according to the first embodiment can perform more accurate determinations based on the "information loss rate of the subject" than conventional cameras and image processing software, which perform blown-out highlight detection based on the entire image. For example, the two-stage blown-out highlight detection unit 208 can detect localized blown-out highlights (e.g., a condition in which light reflects off the surface of a vinyl-coated card, making part of the card information unreadable) that are difficult to detect using existing automatic exposure control algorithms, based on global and local analysis. When the two-stage blown-out highlight detection unit 208 detects blown-out highlights, it outputs information to the display processing unit 203 instructing the user to modify the shooting conditions. The display processing unit 203 then displays information instructing the user to change the shooting position, etc. This ensures stable capture of images without blown-out highlights. In particular, the authenticity determination unit 210, which uses the results of the two-stage blown-out highlight detection unit 208 to determine the authenticity of card media, provides greater stability and accuracy in authenticity determination for subject authentication and check-in applications than conventional methods.

[0179] In addition, camera devices and control software for general smartphones are equipped with an automatic exposure control function as standard, making it difficult to directly acquire raw sensor data. On the other hand, the first embodiment specifically detects only overexposure, which impairs subject information even when using corrected image data, and prompts the user to change the shooting angle appropriately, leading to successful shooting, thereby improving the user experience (UX).

[0180] Furthermore, in this embodiment, the information processing terminal 2 determines the authenticity of the card based on time-series information of the light reflected from the card obtained by the camera 116. This eliminates the need to embed an NFC tag or the like for authenticity determination in the card, or to apply a watermark or special processing for authenticity determination. This not only prevents an increase in the manufacturing cost of the card, but also prevents the aesthetic appeal of the card from being impaired.

[0181] Furthermore, in this embodiment, the photography determination unit 206 performs photography determination to determine whether or not the camera 116 is photographing a physical card medium, using a detection value from the six-axis sensor 117 provided in the information processing terminal 2. Therefore, in this embodiment, it is possible to perform card authenticity determination (photography determination) without introducing new hardware into the information processing terminal 2.

[0182] Furthermore, for example, in authenticating a personal identification number (My Number) card, information on an image of the card is used, so a guide frame or the like cannot be displayed on the screen, but in this embodiment, the authenticity of the card can be determined by inputting a moving image captured by camera 116. Therefore, a guide frame can be displayed in real time on the screen of display device 119.

[0183] Then, the display processing unit 203 moves the position of the guide frame displayed on the screen of the display device 119, thereby allowing the user to perform the operation required for the information processing terminal 2 to acquire the information necessary for determining the authenticity of the card. Furthermore, this operation is a simple operation of moving the information processing terminal 2 so that the frame portion of the card fits into the guide frame, so even users who are unfamiliar with operating the information processing terminal 2 or users who are beginners at trading card games can properly photograph the card.

[0184] In this embodiment, since the authentication determination model 202b constructed by a neural network is used to determine the authenticity of a card, there is a risk that a deceptive signal that exploits the vulnerability of the neural network model using a method not anticipated by the designer may be input during the authentication process. In fact, known attack methods include a method of deceiving a neural network model using a noisy image and a "jailbreak" attack method for circumventing the limitations of large language models (LLMs). However, a method for safely using neural network models has not yet been established.

[0185] In contrast, the authentication system 100 according to this embodiment uses YOLO to detect the position and type of the card, and KAZE to detect the card area. The neural network model used for authentication is highly flexible but behaves non-deterministically. By configuring the authentication system 100 in this way, it becomes possible to determine the authenticity of a card by having the neural network model (authentication model 202b) learn video images of the card, without having the neural network model learn cheating techniques on a large scale.

[0186] As an additional function, the two-stage blown-out highlight detection unit 208 may also perform an RGB saturation check. The RGB saturation check is a method in which the two-stage blown-out highlight detection unit 208 separates the R (Red), G (Green), and B (Blue) channels of the input image, counts the number of pixels whose luminance values ​​in all three channels are equal to or greater than a luminance threshold (e.g., 250), and calculates the ratio of the luminance value of each channel to the overall image. For example, if the two-stage blown-out highlight detection unit 208 cannot correctly detect blown-out highlights in an image captured under lighting of a specific color, performing an RGB saturation check is expected to enable correct detection of blown-out highlights in the image.

[0187] [Second embodiment] <Outline of the authenticity determination system> Next, an authenticity determination system according to a second embodiment of the present invention will be described. In the authenticity determination system according to the second embodiment, the processes related to authenticity determination that were previously performed by the information processing terminal 2 are performed by an inference device on the server side.

[0188] 18 is a diagram showing an overall configuration of an authenticity determination system 100A according to a second embodiment of the present invention, which is a block diagram showing an example of the hardware configuration of a learning device 3, an inference device 6, and an information processing terminal 2A that constitute the authenticity determination system 100A.

[0189] The authenticity determination system 100A includes an information processing terminal 2A and a learning device 3 according to the second embodiment, as well as an inference device 6 that is capable of communicating via a network N. Here, an example configuration of the inference device 6 will be described.

[0190] (Example of inference device configuration) The inference device 6 is an example of a calculator that operates as a computer. The inference device 6 includes a control unit 160, a nonvolatile storage 165, and a network interface 166, all of which are connected to a bus B3.

[0191] The control unit 160 includes a CPU 161, a GPU 162, a ROM 163, and a RAM 164. The control unit 160 may be configured using an FPGA.

[0192] The CPU 161 reads out program code of software that realizes each function according to this embodiment from the ROM 163, loads it into the RAM 164, and executes it. Variables, parameters, etc. that are generated during the calculation processing of the CPU 161 are temporarily written to the RAM 164. These variables, parameters, etc. written to the RAM 164 are read out by the CPU 161 as appropriate. The GPU 162 performs calculations required when performing various inference processes according to this embodiment.

[0193] The nonvolatile storage 165 may be, for example, an HDD, SSD, optical disk, magneto-optical disk, or nonvolatile memory. In addition to an OS and various parameters, this nonvolatile storage 165 stores a program for causing the inference device 6 to function. The program for causing the inference device 6 to function may be stored in ROM 163. In other words, ROM 163 and nonvolatile storage 165 are used as an example of a computer-readable non-transitory recording medium that stores a program executed by the inference device 6. In addition, a server-side inference model DB 60 (see FIG. 6) that stores the authenticity determination model 60b is formed in the nonvolatile storage 165.

[0194] For example, an NIC or the like is used as the network interface 166. The network interface 166 transmits and receives various data to and from the learning device 3 and the information processing terminal 2A via a dedicated line or the like connected to a terminal of the NIC and via the network N.

[0195] <Example of functional configuration of authenticity determination system> Next, with reference to FIG. 19, the functional configuration of the authenticity determining system 100A according to the second embodiment will be described. 19 is a block diagram showing the functional configuration of an authenticity determination system 100A according to the second embodiment. In FIG. 19, the communication network N and the learning device 3 are omitted.

[0196] (Example of terminal device functional configuration) The information processing terminal 2A includes a local image DB 201, a local-side inference model DB 202, a display processing unit 203, a video image capturing unit 204, a card area detection unit 205, a six-axis sensor 117, a capturing determination unit 206, a known item detection unit 207, a two-stage blown-out highlight detection unit 208, and a normalization unit 209. The functions of each unit included in the information processing terminal 2A are the same as those of each unit shown in FIG. 3 according to the first embodiment, and therefore detailed description thereof will be omitted.

[0197] Note that only the known item detection model 202a is stored in the local-side inference model DB 202. When the normalization unit 209 receives an image in which the known item detection unit 207 has detected a known item with a high degree of certainty, or an image in which no blown-out highlights have been detected by the two-stage blown-out highlight detection unit 208, the normalization unit 209 normalizes the image and transmits the image data of the normalized image to the inference device 6. At this time, the image data is transmitted from the network interface 115 of the information processing terminal 2A via the communication network N, and the control unit 160 captures the image data received by the network interface 166 of the inference device 6.

[0198] (Example of functional configuration of inference device) The inference device 6 includes a server-side inference model DB 60 and an authenticity determination unit 61. The server-side inference model DB 60 stores the authenticity determination model 45b imported from the server-side learning model DB 45 of the learning device 3 as the authenticity determination model 60b.

[0199] The authenticity determination unit 61 inputs the image data of the normalized image acquired from the information processing terminal 2A into the authenticity determination model 60b read from the server-side inference model DB 60, and obtains an authenticity determination result. The operation of the authenticity determination model 60b is the same as the authenticity determination process described with reference to Fig. 17. Then, the authenticity determination result is transmitted to the information processing terminal 2A, and the display processing unit 203 displays the authenticity determination result.

[0200] In the authentication determination system 100A according to the second embodiment described above, the authentication determination process is performed by the inference device 6 on the server side. This prevents reverse engineering of the authentication determination model 60b. Furthermore, the authentication determination training data 40c expanded by the data expansion unit 41A is stored in the training data DB 40, preventing an unauthorized user from training the authentication determination model 60b with the same data. This prevents unauthorized modification of the authentication determination model 60b.

[0201] Furthermore, even if an image that has been fraudulently edited by an unauthorized user is detected as a known item by the processing of the information processing terminal 2A and no blown-out highlights are detected in the image, the authenticity can be correctly determined by the authenticity determination unit 61. Because the inference device 6 is managed by the developer, it becomes easy to identify unauthorized users who repeatedly send image data that has been determined to be fake and take measures such as suspending their accounts.

[0202] [Third embodiment] <Outline of the authenticity determination system> Next, an authenticity determination system according to a third embodiment of the present invention will be described. In the authenticity determination system according to the third embodiment, the processes related to known item detection, two-stage overexposure detection, and authenticity determination, which were previously performed by the information processing terminal 2, are performed by an inference device on the server side. Since the hardware configuration of the authenticity determination system according to the third embodiment is the same as that of the authenticity determination system according to the second embodiment, only an example of the functional configuration of the authenticity determination system will be described.

[0203] <Example of functional configuration of authenticity determination system> Next, with reference to FIG. 20, the functional configuration of an authenticity determination system 100B according to the third embodiment will be described. 20 is a block diagram showing the functional configuration of an authenticity determination system 100B according to the third embodiment. In FIG. 20, the communication network N and the learning device 3 are omitted.

[0204] (Example of terminal device functional configuration) The information processing terminal 2B includes a local image DB 201, a display processing unit 203, a moving image shooting unit 204, a card area detection unit 205, a six-axis sensor 117, and a shooting determination unit 206. The functions of the units included in the information processing terminal 2B are the same as the functions of the units shown in Fig. 3 according to the first embodiment, and therefore detailed description thereof will be omitted.

[0205] It should be noted that the information processing terminal 2B itself does not perform AI determination. Image data of the moving image for which the shooting determination unit 206 has performed shooting determination is transmitted to the inference device 6. At this time, the image data is transmitted from the network interface 115 of the information processing terminal 2B via the communication network N, and the control unit 160 captures the image data received by the network interface 166 of the inference device 6A.

[0206] (Example of functional configuration of inference device) The inference device 6A includes a server-side inference model DB 60, a known item detection unit 62, a two-stage overexposure detection unit 63, a normalization unit 64, and an authenticity determination unit 61. The server-side inference model DB 60 stores the known item detection model 45a and the authenticity determination model 45b imported from the server-side learning model DB 45 of the learning device 3 as the known item detection model 60a and the authenticity determination model 60b.

[0207] The known item detection unit 62 inputs the image data obtained from the information processing terminal 2B and after the photographing determination into the known item detection model 60a read from the server-side inference model DB 60, and detects known items from the subjects appearing in the image. The known item detection unit 62 then outputs a certainty score. The processing of the known item detection unit 62 is the same as the processing shown in FIG. 15.

[0208] The two-stage blown-out highlight detection unit 63 performs two-stage blown-out highlight detection processing when the certainty score falls within a range of predetermined thresholds t1 to t2. The processing by the two-stage blown-out highlight detection unit 63 is the same as the processing shown in Fig. 16. When the two-stage blown-out highlight detection unit 63 detects blown-out highlights in the image through global analysis or local analysis, it transmits correction instruction information for the shooting determination conditions to the display processing unit 203 of the information processing terminal 2.

[0209] The normalization unit 64 normalizes an image whose certainty score exceeds t2 or an image in which no blown-out highlights have been detected by the two-stage blown-out highlights detection unit 63. The processing by the normalization unit 64 is the same as the processing in step S38 in FIG.

[0210] As described in the second embodiment, the authenticity determination unit 61 obtains an authenticity determination result based on the image data of the normalized image. The operation of the authenticity determination model 60b is the same as the authenticity determination process described with reference to Fig. 17. Then, the authenticity determination result is transmitted to the information processing terminal 2A, and the display processing unit 203 displays the authenticity determination result.

[0211] In the authenticity determination system 100B according to the third embodiment described above, the inference device 6A on the server side executes processing for not only the authenticity determination model 60b but also the known item detection model 60a. This reduces the processing load of the known item detection process, the two-stage blown-out highlight detection process, the normalization process, and the authenticity determination process on the information processing terminal 2B.

[0212] In addition, the known item detection process, two-stage whiteout detection process, normalization process, and authenticity determination process are performed by the inference device 6 on the server side. This prevents reverse engineering of the known item detection model 60a and the authenticity determination model 60b. Images captured by the information processing terminal 2B are determined in real time by the image capture determination unit 206, and the image data after the image capture determination is acquired in real time by the inference device 6A. The inference device 6A performs the known item detection process, two-stage whiteout detection process, normalization process, and authenticity determination process in real time. Even if an unauthorized user manipulates an image, the developer managing the inference device 6A can understand the manipulation process from the video. This allows unauthorized operations by unauthorized users to be detected and dealt with quickly.

[0213] It should be noted that the present invention is not limited to the above-described embodiments, and various other applications and modifications are possible as long as they do not deviate from the gist of the present invention as set forth in the claims. For example, the above-described embodiments have described the configuration of the device and system in detail and specifically in order to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of the embodiments described here with the configuration of other embodiments, and it is also possible to add the configuration of one embodiment to the configuration of another embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0214] 2...information processing terminal, 3...learning device, 6...inference device, 40...learning data DB, 41, 41A...data expansion unit, 45...server-side learning model DB, 50...master image DB, 60...server-side inference model DB, 100...authenticity determination system, 202a...known item detection model, 202b...authenticity determination model, 203...display processing unit, 204...video image capturing unit, 205...card area detection unit, 206...photography determination unit, 207...known item detection unit, 208...two-stage overexposure detection unit, 209...normalization unit, 210...authenticity determination unit, 201...local image DB, 202...local-side inference model DB, 411...extraction unit, 412...replication unit, 413...expansion unit

Claims

1. A learning program for learning a card medium in which a processing process for changing a reflection state of light reflected from a predetermined surface of the card medium has been applied to the predetermined surface, an extraction step of extracting an arbitrary number of pieces of moving image data from a plurality of pieces of moving image data shot in time series on the card medium, and extracting, for each piece of extracted moving image data, an arrangement of images included in the moving image data as an initial image arrangement; a duplication step in which one of the images selected from the initial image array is duplicated any number of times to generate an array of static images; an extension procedure of copying the array of static images, applying a predetermined extension process to any static images included in the array of static images to generate an array of dynamic images, and recording the array of static images and the array of dynamic images in a learning data recording unit as authenticity determination learning data for learning the card medium; an authenticity determination learning procedure for learning the card medium into an authenticity determination model into which the initial image array, the static image array, and the dynamic image array have been read, and recording the learned authenticity determination model in a learning model recording unit; A learning program for a computer to run.

2. The extension procedure generates the sequence of dynamic images by an extension process that combines an independent image process that performs arbitrary image processing on one or more static images extracted from the sequence of static images, and a continuous image process that performs the arbitrary image processing on a plurality of the static images extracted from the sequence of static images. The learning program according to claim 1 .

3. The optional image processing includes at least one of adding blur to the static image, cropping a rectangular area from the static image, enlarging, shrinking, rotating, or translating the static image, changing the brightness of a portion of the static image, and cropping the edges of the static image. The learning program according to claim 2 .

4. The independent image processing performs a geometric transformation process on the static images randomly extracted from the array of static images. The learning program according to claim 3 .

5. A learning method for learning a card medium in which a processing process is applied to a predetermined surface of the card medium to change the reflection state of reflected light from the predetermined surface, an extraction step of extracting an arbitrary number of pieces of moving image data from a plurality of pieces of moving image data shot in time series on the card medium, and extracting, for each piece of extracted moving image data, an arrangement of images included in the moving image data as an initial image arrangement; a duplication step in which one of the images selected from the initial image array is duplicated any number of times to generate an array of static images; an extension procedure of copying the array of static images, applying a predetermined extension process to any static images included in the array of static images to generate an array of dynamic images, and recording the array of static images and the array of dynamic images in a learning data recording unit as authenticity determination learning data for learning the card medium; an authenticity determination learning procedure for learning the card medium into an authenticity determination model into which the initial image array, the static image array, and the dynamic image array have been read, and recording the learned authenticity determination model in a learning model recording unit. How to learn.

6. A learning device for learning a card medium in which a processing process is applied to a predetermined surface of the card medium to change the reflection state of reflected light from the predetermined surface, an extracting unit that extracts an arbitrary number of pieces of moving image data from a plurality of pieces of moving image data captured in time series on the card medium, and extracts, for each piece of extracted moving image data, an arrangement of images included in the moving image data as an initial image arrangement; a replicating unit that replicates one of the images selected from the initial image array an arbitrary number of times to generate an array of static images; an expansion unit that copies the array of static images, performs a predetermined expansion process on any static images included in the array of static images to generate an array of dynamic images, and records the array of static images and the array of dynamic images in a learning data recording unit as authenticity determination learning data for learning the card medium; an authenticity determination learning unit that causes an authenticity determination model into which the initial image array, the static image array, and the dynamic image array are read to learn the card medium, and records the learned authenticity determination model in a learning model recording unit. Learning device.

7. An authenticity determination program for determining the authenticity of a card medium having a predetermined surface to which processing has been applied to change the reflection state of light reflected from the predetermined surface, a display control procedure for displaying a guide for changing the shooting angle of a moving image of the card medium captured by a camera provided in a terminal device on a screen of the terminal device, and changing the display position of the guide during a shooting period; a known image detection step of detecting a known image formed on the card medium from the moving image data of the card medium captured according to the guide; When the known image is detected, a trained authenticity determination model is input with second moving image data of the card medium captured by the camera while the guide is displayed, and read from a recording unit of the terminal device, the trained authenticity determination model including an extraction step of extracting an arbitrary number of moving image data from a plurality of moving image data captured in time series of the card medium, and extracting, for each of the extracted moving image data, an array of images included in the moving image data as an initial image array; a duplication step of duplicating an arbitrary number of images selected from the initial image array to generate an array of static images; and a duplication step of copying the array of static images and extracting the images included in the array of static images to be copied. an authenticity determination learning procedure for causing the card medium to learn into the authenticity determination model into which the initial image array, the static image array, and the dynamic image array have been read, by a learning program for causing a computer to execute the above steps; and a determination procedure for determining the authenticity of the card medium based on time series information of reflected light from a predetermined surface of the card medium using the trained authenticity determination model that has trained the card medium using the training program. An authenticity determination program to be run by a computer.

8. A method for determining the authenticity of a card medium having a predetermined surface to which a processing treatment for changing a reflection state of reflected light from the predetermined surface has been applied, comprising: a display control procedure for displaying a guide for changing the shooting angle of a moving image of the card medium captured by a camera provided in a terminal device on a screen of the terminal device, and changing the display position of the guide during a shooting period; a known image detection step of detecting a known image formed on the card medium from the moving image data of the card medium captured according to the guide; When the known image is detected, a trained authenticity determination model is input with second moving image data of the card medium captured by the camera while the guide is displayed, and read from a recording unit of the terminal device, the trained authenticity determination model including: an extraction step of extracting an arbitrary number of moving image data from a plurality of moving image data captured in time series of the card medium, and extracting, for each of the extracted moving image data, an arrangement of images contained in the moving image data as an initial image arrangement; a duplication step of duplicating an arbitrary number of images selected from the initial image arrangement to generate an arrangement of static images; and a duplication step of copying the arrangement of the static images to generate the static images as the copy destinations. an authenticity determination learning procedure that learns the card medium using the authenticity determination model that has learned the card medium using a learning method including: an extension procedure that performs a predetermined extension process on any static image included in the image array to generate an array of dynamic images, and records the array of static images and the array of dynamic images in a learning data recording unit as authenticity determination learning data for learning the card medium; and an authenticity determination learning procedure that causes the authenticity determination model that has read the initial image array, the array of static images, and the array of dynamic images to learn the card medium, and a determination procedure that determines the authenticity of the card medium based on time-series information of reflected light from a predetermined surface of the card medium using the trained authenticity determination model that has learned the card medium using a learning method including: Authenticity determination method.

9. An authenticity determination device for determining the authenticity of a card medium having a predetermined surface that has been subjected to a processing treatment that changes the reflection state of reflected light from the predetermined surface, a display control unit that displays a guide for changing the shooting angle of the moving image of the card medium captured by a camera provided in the terminal device on a screen of the terminal device and changes the display position of the guide during the shooting period; a known image detection unit that detects a known image formed on the card medium from the moving image data of the card medium captured according to the guide; When the known image is detected, a trained authenticity determination model is input with second moving image data of the card medium captured by the camera while the guide is displayed, and read from a recording unit of the terminal device, the trained authenticity determination model comprising: an extracting unit that extracts an arbitrary number of moving image data from a plurality of moving image data captured in time series of the card medium, and extracts, for each of the extracted moving image data, an arrangement of images included in the moving image data as an initial image arrangement; a duplicating unit that copies one image selected from the initial image arrangement by an arbitrary number to generate an arrangement of static images; and a copying unit that copies the arrangement of the static images and applies a predetermined value to an arbitrary static image included in the arrangement of the static images to which the images have been copied. an authenticity determination learning unit that causes the authenticity determination model that has been trained with the card medium to learn the card medium, the authenticity of the card medium being determined based on time-series information of light reflected from a predetermined surface of the card medium; and a determination unit that determines the authenticity of the card medium based on time-series information of light reflected from a predetermined surface of the card medium. Authenticity determination device.

Images