Inference program, inference method, inference device, learning program, learning method, and learning device

Abstract

An inference device is provided that can correctly identify a plurality of objects appearing in a captured image and determine whether authentication has succeeded or failed. [Solution] The inference device 60 includes an inference device-side communication unit 91 that acquires a captured image 83 in which multiple objects are captured and a surrounding image identification area set in part of the captured image 83 for identifying the surrounding image of the object, and an object identification AI model 43 that sets an object detection area in which the object is detected for each of the multiple objects captured in the captured image 83 and identifies the object for each object detection area, and determines whether user authentication has been successful or unsuccessful based on the position of the object detection area relative to the surrounding image identification area.

Description

[Technical Field]

[0001] The present invention relates to an inference program, an inference method, an inference device, a learning program, a learning method, and a learning device. [Background technology]

[0002] Conventionally, there has been provided a reward system that identifies a single product from a photographed image, and if the photographed product is deemed appropriate, grants a reward (including rewards such as points, in-game currency, in-game items, etc.) to the user who sent the photographed image. Since a reward is granted to a user whose photographed product is determined to be appropriate, it is expected that the user's desire to purchase the product will increase.

[0003] Patent Document 1 describes a system including a terminal device and a server. The terminal device described in Patent Document 1 determines whether any of multiple types of objects is captured in an image of space captured by an imaging device of the mobile terminal device based on local feature values ​​for shapes used to detect multiple types of objects. If it is determined that any type of object is captured, the image determined to include the object is captured as an acquired object image and transmits the acquired object image to a server. The server receives the transmitted acquired object image and determines a feature correlation value of the acquired object image with respect to one of the multiple types of objects based on the image feature value of the one type of object, and determines whether the acquired object image is an image of the one type of object based on the determined feature correlation value. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6517298 Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional reward systems assume that a single product appears in a captured image. If the results of image processing and object recognition performed on the captured image are valid, the system determines that the user has been successfully authenticated and awards a reward to the user. However, conventional reward systems cannot identify products when targeting multiple products (e.g., a burger and a drink combination), such as a set menu at a restaurant or food delivery service. Set menus often include multiple types of drinks and burgers, and many combinations. Furthermore, it is necessary to assume that users may arrange the products in any order and photograph them at any angle. For this reason, image recognition technology for a single object, such as that disclosed in Patent Document 1, cannot be used at all.

[0006] Furthermore, a new problem has arisen in photographed images of multiple products: occlusions, such as a burger appearing in front of a drink cup and hiding part of the cup, or conversely, a drink appearing in front of a burger and hiding part of the burger. This has led to a demand for a system that can identify multiple set menu products appearing in a photographed image in a predetermined combination and reliably authenticate a user, and there has been a need for a system that can correctly identify sets of products.

[0007] Furthermore, in a reward system that rewards users who submit images of set menus taken at any location, it is also necessary to deal with fraudulent user manipulation. For example, one example of fraudulent manipulation is an attempt to successfully authenticate someone by displaying an image of a set menu (e.g., an image obtained from the Internet) on the display of a smartphone or personal computer (PC), and then using a photograph of this displayed image.

[0008] The present invention has been made in view of the above circumstances, and aims to correctly identify multiple objects appearing in a captured image, detect fraud, and determine whether authentication has succeeded or failed. [Means for solving the problem]

[0009] The inference program according to the present invention includes the steps of: acquiring a photographed image in which a plurality of objects are photographed; and acquiring area information of a surrounding image identification area set in a portion of the photographed image for identifying surrounding images of the objects; and setting an object detection area in which an object identification model detects each of the plurality of objects appearing in the photographed image, and identifying the plurality of objects appearing in the photographed image for each object detection area. When the overlap between the object detection area and the surrounding image identification area is equal to or greater than a predetermined amount, Authenticating Users is determined to be a failure, and if the overlap between the object detection area and the surrounding image identification area is less than a predetermined amount and the group of identified objects does not satisfy the grouping condition, the first determination performed for authentication is determined to be a failure, and if the overlap between the object detection area and the surrounding image identification area is less than a predetermined amount and the group of identified objects satisfies the grouping condition, the first determination is determined to be a success. and causing the computer to execute the steps of: The above-described inference program is one aspect of the present invention, and an inference method and an inference device that reflect one aspect of the present invention are configured in the same manner as the above-described inference program. [Effects of the Invention]

[0011] According to the present invention, it is possible to correctly identify multiple objects captured in a captured image and detect fraud, thereby determining whether authentication has succeeded or failed, thereby preventing fraud. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is an overall configuration diagram showing an overview of a reward granting system according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of an information processing terminal according to a first embodiment of the present invention. [Figure 3] 1 is a block diagram showing a hardware configuration of a learning device according to a first embodiment of the present invention. [Figure 4] 1 is a block diagram showing an example of the functional configuration of a learning device according to a first embodiment of the present invention. [Figure 5] 5 is a flowchart illustrating an example of an object identification learning process according to the first embodiment of the present invention. [Figure 6] 1 is a flowchart illustrating an example of an image classification learning process according to the first embodiment of the present invention. [Figure 7]1 is a block diagram showing the hardware configuration of an inference device according to a first embodiment of the present invention. [Figure 8] 3A to 3C are diagrams showing examples of a shooting area, an internal area, and a surrounding image identification area according to the first embodiment of the present invention. [Figure 9] FIG. 10 is a diagram showing an example of a check-in screen displayed when check-in is successful according to the first embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing an example of a check-in screen displayed when check-in has failed according to the first embodiment of the present invention. [Figure 11] 1 is a block diagram showing an example of the functional configuration of a reward granting system according to a first embodiment of the present invention. [Figure 12] FIG. 3 is a sequence diagram showing an example of processing by the information processing terminal and the inference device according to the first embodiment of the present invention. [Figure 13] 10 is a flowchart illustrating an example of an object identification inference process according to the first embodiment of the present invention. [Figure 14] FIG. 3 is a diagram showing an example of an object identification result according to the first embodiment of the present invention. [Figure 15] 5 is a flowchart illustrating an example of an object identification and determination process according to the first embodiment of the present invention. [Figure 16] 5A to 5C are diagrams showing examples of object identification results and check-in results according to the first embodiment of the present invention. [Figure 17] 4 is a flowchart illustrating an example of an image classification inference process according to the first embodiment of the present invention. [Figure 18] FIG. 3 is a diagram showing an example of the configuration of a determination result table according to the first embodiment of the present invention. [Figure 19] FIG. 10 is a block diagram showing an example of the functional configuration of a reward granting system according to a second embodiment of the present invention. [Figure 20] FIG. 10 is a block diagram showing an example of the functional configuration of a reward granting system according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted.

[0014] [First embodiment] Before determining whether a reward (including points, in-game currency, in-game items, and other benefits) is awarded to a user who has photographed an existing product with a camera, the process of determining the correct combination of objects from the captured image using object recognition (first determination) and the process of detecting that the captured image is not fraudulent (second determination) are combined to authenticate the user, which is called check-in. A system in which a reward is awarded to a user authenticated by check-in is called a reward awarding system. For this reason, a reward awarding system is also called a check-in system. In a reward awarding system, a user who performs fraudulent operations is determined to have failed authentication, and a reward is not awarded to the user, or any reward that has already been awarded is canceled. The realization of such a reward awarding system is essentially important for promoting products and / or services.

[0015] Previously, user authentication was determined using methods such as two-dimensional codes or near-field radio communication (NFC). However, these methods required modifications to the POS (Point of Sale) system, which was very costly to implement. There are also systems that use Bluetooth Low Energy (BLE) beacons to authenticate users, which do not require modifications to the POS system. However, small devices that emit BLE beacons can only be used to check in at specific locations, and deploying them across thousands of stores tends to be expensive, making it difficult to confirm that a user has purchased a specific product.

[0016] On the other hand, the reward granting system according to the first embodiment of the present invention is a technology that targets a set menu that combines predetermined products and verifies that a user has purchased the products by taking a photo of the set of purchased products with a smartphone camera. In the embodiment described below, a reward granting system is described that has the ability to identify the type of product, in particular, a set menu consisting of a combination of a specific variety of products (an example of an object), and the tamper resistance to prevent fraudulent authentication.

[0017] <Example of overall configuration of reward system> First, we will explain an example of the configuration of a reward granting system according to the first embodiment of the present invention. This reward granting system is configured by combining a learning device that learns object identification processing and image classification processing, an inference device that infers object identification and image classification, and an information processing terminal. 1 is a diagram showing an overall configuration of a reward granting system 10 according to a first embodiment of the present invention. In the following, an object that is the target of object identification processing or the like may be described as a commodity.

[0018] The reward granting system 10 includes a tablet terminal 2_1, a PC (Personal Computer) 2_2, a learning device 30, and an inference device 60. The tablet terminal 2_1 and PC 2_2 used by users can be connected to the inference device 60 via a network N such as the Internet. The tablet terminal 2_1 and PC 2_2 used by an administrator can be connected to the learning device 30 and the inference device 60. In the following description, the tablet terminal 2_1 and PC 2_2 are collectively referred to as information processing terminal 2.

[0019] The learning device 30 is a device that trains an object identification AI model 43 (see FIG. 4 described later), which is an example of an object identification model, to perform object identification processing, and trains an image classification AI model 53 (see FIG. 4 described later) to perform image classification processing. The inference device 60 is a device that trains the trained object identification AI model 43 to identify products appearing in photographed images, and trains the trained image classification AI model 53 to perform binary classification of the photographed images as positive examples or negative examples.

[0020] For this reason, the learning device 30 and the inference device 60 manage programs and various data used as the object identification AI model 43 and the image classification AI model 53. YOLO (You Only Look Once), which will be described later, is used as the object identification AI model 43. ResNet, which will be described later, is used as the image classification AI model 53.

[0021] The following description is given assuming that the information processing terminal 2 is operated by a user. The user is assumed to be a person who purchases a set of products, takes a photo of the set of products, and operates the information processing terminal 2 to request authentication.

[0022] The tablet terminal 2_1 constituting the information processing terminal 2 uses a touch panel display device in which the input device 26 and the output device 27 are integrated. The PC 2_2 has the input device 26 and the output device 27 as separate devices. Note that the PC 2_2 may be configured as a desktop PC, with the input device 26 and the output device 27 separately connected to the desktop PC.

[0023] The information processing terminal 2 selects a program based on an operation signal input from the input device 26 in response to an operation performed by the user, and outputs a video signal corresponding to the screen of the output device 27 to the output device 27. The output device 27 displays an image based on the video signal. The operation signal input from the input device 26 is, for example, a signal corresponding to each operation button on a keyboard. The user can input instructions through the input device 26, instruct the inference device 60 to execute an inference program, or operate an inference program recorded in the recording device 22 of the user's terminal. Operations on the inference program input from the input device 26 include, for example, various command inputs such as starting an application program, performing a photographing operation, and instructing the user to send a photographed image. Another example of an operation performed from the input device 26 is a tap operation, such as touching the screen of the output device 27 with a finger or a pen.

[0024] The information processing terminal 2 performs processes such as reading image data from the recording device 22 and executing a program, and displaying a screen on the output device 27 in accordance with an operation signal input from the input device 26. For example, in accordance with an operation by a user via the input device 26, the information processing terminal 2 displays on the output device 27 a screen on which an image based on the image data read from the recording device 22 is drawn.

[0025] During learning, the information processing terminal 2 is used by the administrator. An operational input is input to the information processing terminal 2 to send captured images of multiple products in a predetermined combination for object identification learning to the learning device 30, causing the object identification AI model 43 to learn to identify the products for object identification. In addition, an operational input is input to the information processing terminal 2 to send captured images for image classification, in which multiple products are shown in a predetermined combination, to the learning device 30, causing the image classification AI model 53 to learn to classify the captured images as positive examples or negative examples. In addition, the information processing terminal 2 is also capable of performing data expansion on captured images for object identification learning.

[0026] The inference program according to this embodiment, which runs on the inference device 60, causes the object recognition AI model 43 to identify multiple products shown in a captured image 83 based on instruction information input via the input device 26 (an example of an input unit). The inference program according to this embodiment also causes the image classification AI model 53 to classify the captured image into a positive example or a negative example.

[0027] <Example of hardware configuration for reward granting system> Next, an example of the hardware configuration of each terminal and device included in the reward granting system 10 according to the first embodiment will be described. 2 is a block diagram showing an example of the hardware configuration of the information processing terminal 2. Examples of the hardware configuration of the learning device 30 and the inference device 60 will be described later.

[0028] (Example of information processing terminal configuration) The information processing terminal 2 is an example of a computer that operates as a computer capable of executing various programs. The information processing terminal 2 includes a processor 21, a recording device 22, and a network interface 24, all of which are connected to a bus 23.

[0029] The processor 21 is configured with at least one of, for example, a central processing unit (CPU), a microprocessor unit (MPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA). The processor 21 reads program code of an application that realizes each function according to this embodiment from the recording device 22, loads the program code into a temporary storage unit (not shown) provided in the recording device 22, and executes the program code. The processor 21 performs, for example, arithmetic processing based on information acquired from the inference device 60, and processing required to draw a GUI of the application on the output device 27 of the information processing terminal 2. The processor 21 also performs processing such as processing the operating system (OS) of the information processing terminal 2 and managing the input and output of data performed by each unit in the information processing terminal 2. When processing information related to user authentication, the processor 21 can output an image signal to the output device 27 through the input / output interface 25.

[0030] The recording device 22 is configured by, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM may be an optical disk, a magneto-optical disk, a DVD (Digital Versatile Disc)-ROM, a CD-ROM, a Blu-ray (registered trademark) disk, or the like. The RAM may be an SRAM (Static RAM), a DRAM (Dynamic RAM), or the like. Variables, parameters, and the like generated during the arithmetic processing of the processor 21 are temporarily written to the recording device 22, and these variables, parameters, and the like are read out by the processor 21 as appropriate.

[0031] The recording device 22 is configured by at least one of, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and a flash memory. The recording device 22 stores the OS of the information processing terminal 2, various parameters, programs for operating the information processing terminal 2, application programs used for user authentication, and the like. As described above, the recording device 22 stores programs, data, and the like necessary for the processor 21 to operate, and is used as an example of a computer-readable non-transitory storage medium that stores programs executed by the information processing terminal 2.

[0032] For example, a network interface card (NIC) or the like is used as the network interface 24. The network interface 24 can transmit and receive various data to and from the learning device 30 and the inference device 60 via a dedicated line or the like connected to a terminal of the NIC and via the network N, and can also communicate with other information processing terminals 2.

[0033] The input / output interface 25 converts operation signals received from the input device 26 into data in a predetermined format and passes the converted data to the processor 21. The input / output interface 25 also converts screen data drawn by the processor 21 into video signals and outputs them to the output device 27.

[0034] The input device 26 is a device that accepts input instructions or various types of information from a user. An example of the input device 26 is a pointing device that can input coordinate information of a position designated by the user. This pointing device is a mouse, a touch panel device, or the like. A touch panel device is configured by combining the input device 26 and the output device 27. The input device 26 may also be a keyboard, a mouse, or the like.

[0035] The output device 27 is a device that outputs information processed by the processor 21. The output device 27 is, for example, a display device (a display device, a touch panel device, etc.). When the output device 27 is a display device, an image (for example, a captured image) based on a video signal received from the input / output interface 25 is displayed on the display device.

[0036] Camera 28 can capture various images in response to operations by the user of information processing terminal 2. Camera 28 stores still images as captured images in recording device 22. Camera 28 can also store moving images as captured images in recording device 22. When capturing an image using camera 28, an image of the product is displayed on output device 27, allowing the user to confirm the position where the product will appear in the captured image.

[0037] <Example of hardware configuration for learning device> Next, an example of the hardware configuration of the learning device 30 will be described. 3 is a block diagram showing the hardware configuration of a learning device 30 according to a first embodiment of the present invention. The learning device 30 is an example of a system for generating an object identification AI model 43 and an image classification AI model 53 (see FIG. 4, both of which will be described later), which are trained models configured to include one or more devices. However, in the following embodiments, for convenience of explanation, the learning device 30 will be described as a single device. The system for generating the object identification AI model 43 and the image classification AI model 53 can also refer to the learning device 30. The same applies to an inference device 60, which will be described later.

[0038] The learning device 30 includes a processor 31, an input device 32, a display device 33, a recording device 34, and a communication device 35. These components are connected by a bus 36. Note that an interface is interposed between the bus 36 and each component device as necessary. The learning device 30 includes a configuration similar to that of a general server, PC, etc.

[0039] The processor 31 controls the overall operation of the learning device 30. For example, the processor 31 is at least one of a CPU, an MPU, a GPU, and an FPGA. The processor 31 performs various processes by reading and executing programs and data stored in the recording device 34. The processor 31 may be composed of multiple processors.

[0040] Input device 32 is a user interface that accepts input from the user to study device 30, and is, for example, a touch panel, touchpad, keyboard, mouse, or button. Display device 33 is a display that displays application screens and the like to the user of study device 30 under the control of processor 31.

[0041] The recording device 34 (an example of a recording unit) includes a main memory device and an auxiliary memory device. The main memory device is, for example, a semiconductor memory such as RAM. RAM is a volatile storage medium that allows high-speed reading and writing of information, and is used as a storage area and a working area when the processor 31 processes information. The main memory device may also include ROM, which is a read-only nonvolatile storage medium. The auxiliary storage device stores various programs and data used by the processor 31 when executing each program. The auxiliary storage device may be any nonvolatile storage or nonvolatile memory that can store information, and may be removable.

[0042] The communication device 35 transmits and receives data to and from the information processing terminal 2 or other computers such as a server via a network, and is, for example, a wireless LAN module. The communication device 35 may be a device or module for other wireless communication such as a Bluetooth (registered trademark) module, or may be a device or module for wired communication such as an Ethernet (registered trademark) module or a USB interface. The system configuration and data structure of this embodiment will be described in detail below.

[0043] <Example of functional configuration of learning device> Next, the process of generating the object identification AI model 43 and the image classification AI model 53 will be described. 4 is a block diagram showing an example of the functional configuration of a learning device 30 according to a first embodiment of the present invention. The learning device 30 includes an object identification learning unit 42, an object identification AI model conversion unit 44, an image classification learning unit 52, and an image classification AI model conversion unit 54. The learning device 30 also includes object identification learning data 41, an object identification AI model 43, a terminal object identification AI model 45, image classification learning data 51, an image classification AI model 53, and a terminal image classification AI model 55, all of which are recorded in a recording device 34.

[0044] (Object discrimination learning) First, an example of the configuration and processing related to object identification learning will be described. The object identification learning unit 42 reads the object identification learning data 41 prepared in the recording device 34 shown in FIG. 3 and generates an object identification AI model 43 that has been trained to perform object identification processing. The object identification processing is a process of identifying multiple products that appear in a captured image. The object identification learning data 41 stores a large number of captured images for object identification learning. The captured image shows multiple products in a predetermined combination, and each product is pre-labeled with a label indicating the product name. The object identification AI model 43 generated by the object identification learning unit 42 is saved in the recording device 34. When different object identification learning data 41 is prepared, the object identification learning unit 42 can have the object identification AI model 43 read from the recording device 34 train the object identification processing again.

[0045] Returning to the explanation of Figure 4. The object identification AI model 43 described above is configured, for example, using YOLO. YOLO is a deep learning model capable of real-time processing used for object detection. YOLO's greatest feature is its extremely high speed and high accuracy in object detection, achieved by performing end-to-end inference on object detection and labeling. YOLO also captures background patterns and object features as a whole, resulting in fewer false positives and higher generalization performance than other methods. In particular, even when a product is placed in front of another product, causing occlusion, YOLO can correctly identify the object as long as the object features partially match.

[0046] We will now explain the YOLO training method. To train YOLO, a "web app for taking training data" implemented using the same code as the prototype web application is used to take photos of set menu items, and the areas of the photographed items are selected and labeled to prepare data. At this time, the training data is reduced and cropped to the same aspect ratio as during inference.

[0047] The object identification AI model conversion unit 44 generates a terminal-use object identification AI model 45 by optimizing the object identification AI model 43 read from the recording device 34. The terminal-use object identification AI model 45 uses less memory than the original object identification AI model 43, and the size of the model file itself is also reduced. This makes it possible to operate the terminal-use object identification AI model 45 even on an information processing terminal 2 with limited resources. The terminal-use object identification AI model 45 is used in the second and third embodiments described below.

[0048] FIG. 5 is a flowchart showing an example of the object identification learning process. First, the object identification learning unit 42 acquires object identification learning data 41 from the recording device 34 (S1). Next, the object identification learning unit 42 generates an object identification AI model 43 that has been trained to perform object identification processing based on the object identification learning data 41 (S2). Next, the object identification learning unit 42 records the trained object identification AI model 43 in the recording device 34 (S3). If the terminal object identification AI model 45 is not to be used, this process ends.

[0049] When using the object identification AI model 45 for terminal, the object identification AI model conversion unit 44 converts the object identification AI model 43 read from the recording device 34 into the object identification AI model 45 for terminal (S4). Next, the object identification AI model conversion unit 44 records the object identification AI model 45 for terminal in the recording device 34 (S5), and this process ends.

[0050] (Image classification learning) Next, an example of the configuration and processing related to image classification learning will be described. In the past, there have been cases of fraudulent attempts to successfully authenticate a reward system by displaying or printing a photo obtained from the Internet or the like without purchasing a product. Since it is necessary to detect such fraud, in this embodiment, an image classification AI model 53 capable of classifying images into positive examples or negative examples is constructed as an example of an image classification model.

[0051] The image classification learning unit 52 reads image classification learning data 51 prepared in the recording device 34 and generates an image classification AI model 53 that has been trained to perform image classification processing. The image classification processing is a process of classifying captured images for image classification, which show multiple products in a predetermined combination, into positive examples or negative examples. The image classification learning data 51 stores a large number of captured images for image classification learning. The captured images show multiple products in a predetermined combination, with captured images for image classification learning being positive examples and fraudulent images being negative examples. The image classification AI model 53 generated by the image classification learning unit 52 is saved in the recording device 34. When different image classification learning data 51 is prepared, the image classification learning unit 52 can train the image classification AI model 53 read from the recording device 34 to perform image classification processing again.

[0052] The positive example captured images include object identification learning data 41. Although negative example captured images are not shown, they may be, for example, images captured by processing another captured image with a paint tool, printed images, etc.

[0053] The image classification AI model 53 is configured, for example, by ResNet. ResNet is not specialized for a specific purpose, but is a model that exhibits extremely high performance in image classification. It is characterized by its ability to improve performance even with very deep networks (e.g., 700 layers) and achieve high accuracy in image recognition tasks. ResNet training is extremely simple, and it learns binary classification processing using the data used in the training of the above-mentioned YOLO as positive examples and images of fraudulent check-ins intentionally created manually by debugging staff, for example, as negative examples.

[0054] The image classification AI model conversion unit 54 generates a terminal-use image classification AI model 55 by optimizing the image classification AI model 53 read from the recording device 34. The terminal-use image classification AI model 55 uses less memory than the original image classification AI model 53, and the size of the model file itself is also reduced. Therefore, the terminal-use image classification AI model 55 can be operated even on an information processing terminal 2 with limited resources. The terminal-use image classification AI model 55 is used in a third embodiment, which will be described later.

[0055] The object identification AI model conversion unit 44 and image classification AI model conversion unit 54 each have the function of converting AI models into a highly efficient neural network representation file format called ONNX, thereby reducing file size. The terminal object identification AI model 45 and terminal image classification AI model 55 are small enough to be loaded onto the RAM or VRAM of the information processing terminal 2, making deployment on the information processing terminal 2 a realistic option. Furthermore, the terminal object identification AI model 45 and terminal image classification AI model 55 perform inference processing on middleware called ONNX Runtime, which optimizes and executes neural network models, and their behavior is automatically optimized for the SIMD instructions and GPU instructions of the CPU of the information processing terminal 2, enabling inference at realistic speeds.

[0056] FIG. 6 is a flowchart illustrating an example of an image classification learning process. First, image classification learning unit 52 acquires image classification training data 51 from recording device 34 (S11). Next, image classification learning unit 52 generates image classification AI model 53 that has been trained to perform image classification processing based on image classification training data 51 (S12). Next, image classification learning unit 52 records trained image classification AI model 53 in recording device 34 (S13). If terminal image classification AI model 55 is not to be used, this process ends.

[0057] When using the terminal-use image classification AI model 55, the image classification AI model conversion unit 54 converts the image classification AI model 53 read from the recording device 34 into the terminal-use image classification AI model 55 (S14). Next, the image classification AI model conversion unit 54 records the terminal-use image classification AI model 55 in the recording device 34 (S15), and the process ends.

[0058] (Example of hardware configuration of inference device) Next, an example of the configuration of the inference device 60 will be described. 7 is a block diagram showing the hardware configuration of an inference device 60 according to a first embodiment of the present invention. The inference device 60 includes a processor 61, an input device 62, a display device 63, a recording device 64, and a communication device 65. These components are connected by a bus 66. Note that an interface is interposed between the bus 66 and each component device as necessary. The inference device 60 includes a configuration similar to that of a general server, PC, etc.

[0059] The processor 61 controls the overall operation of the inference device 60. For example, the processor 61 is at least one of a CPU, an MPU, a GPU, and an FPGA. The processor 61 performs various processes by reading and executing programs and data stored in the recording device 64. The processor 61 may be composed of multiple processors.

[0060] The input device 62 is a user interface that accepts input from a user to the inference device 60, and is, for example, a touch panel, a touch pad, a keyboard, a mouse, or a button. The display device 63 is a display that displays application screens and the like to the user of the inference device 60 under the control of the processor 61.

[0061] The recording device 64 includes a main memory device and an auxiliary memory device. The main memory device is, for example, a semiconductor memory such as RAM. RAM is a volatile storage medium that allows high-speed reading and writing of information, and is used as a storage area and a working area when the processor 61 processes information. The main memory device may also include ROM, which is a read-only nonvolatile storage medium. The auxiliary storage device stores various programs and data used by the processor 61 when executing each program. The auxiliary storage device may be any nonvolatile storage or nonvolatile memory that can store information, and may be removable.

[0062] The communication device 65 transmits and receives data to and from the information processing terminal 2 or other computers such as a server via a network, and is, for example, a wireless LAN module. The communication device 65 may be a device or module for other wireless communication such as a Bluetooth (registered trademark) module, or may be a device or module for wired communication such as an Ethernet (registered trademark) module or a USB interface.

[0063] <Functional explanation of the inference device> Next, the functions of the inference device 60 will be described. The inference device 60 is composed of a user interface part (referred to as a UI part) that acquires check-in images, and a back-end part (referred to as a BE part) that verifies the check-in images using two systems of AI models. First, the user interface part that the user sees directly will be described. The check-in user interface and each area in this embodiment will be described with reference to Figs. 8 to 10.

[0064] <Explanation of each area> First, with reference to FIG. 8, each area displayed on the display device (output device 27) of the information processing terminal 2 will be described. FIG. 8 is a diagram showing an example of a photographing area, an internal area, and a surrounding image identification area.

[0065] The photographic area display unit 81 represents the screen displayed on the information processing terminal 2 used in portrait orientation. The photographable area 101 is the area surrounded by a dashed line in the figure, and is the area that can be photographed by the camera 28. The display area 102 is the area hatched in the figure. The display device displays an image that appears in the display area 102, out of the area that can be photographed in the photographable area 101.

[0066] The photographing area 103 is an area surrounded by a dashed line in the figure, and indicates the size of the photographed image. The product photographed by the camera 28 is recorded in the recording device 22 shown in Fig. 2 as a photographed image 83 of the size displayed in the photographing area 103. The aspect ratio of the photographing area 103 is set to, for example, 10:13.

[0067] The internal region 104 is an internal region obtained by reducing the imaging region 103 with the center point of the imaging region 103 as a reference. The area of ​​the internal region 104 is adjusted to be, for example, 70% or less of the area of ​​the imaging region 103. The position of the internal region 104 is specified by the x and y coordinates of the upper left corner, and the size of the internal region 104 is specified by a height h and a width w. The portion of the imaging region 103 excluding the internal region 104 is called a peripheral image identification region 105. The area of ​​the peripheral image identification region 105 is smaller than the area of ​​the imaging region 103. For example, the area of ​​the peripheral image identification region 105 is adjusted to be more than 30% of the area of ​​the imaging region 103.

[0068] 9 is a diagram showing an example of the check-in screen W1 displayed when check-in is successful. The check-in screen W1 is displayed on the output device 27 of the information processing terminal 2, and is a screen on which the success or failure of check-in is output.

[0069] Display example (1) in FIG. 9 shows an example of a check-in screen W1 in which an application installed on information processing terminal 2 activates camera 28 and displays an image from camera 28 on the page in real time. Check-in screen W1 displays a bounding box 106, which is a circular boundary line. Bounding box 106 is configured as a rectangle and indicates the boundary between internal area 104 and surrounding image identification area 105. In addition, a message display area 107 shown at the bottom of check-in screen W1 displays a message instructing the user to fit the product to be checked in within bounding box 106.

[0070] The brightness of the peripheral image identification area 105 is displayed lower than the brightness of the internal area 104, which is inside the boundary line representing the boundary frame 106. For this reason, the user attempts to operate the camera 28 to capture an image so that the product is captured inside the boundary frame 106. As a result, the camera 28 captures not only an image captured in the internal area 104, but also an image of the peripheral image identification area 105, which is outside the boundary frame 106. Note that the diagonal line crossing the internal area 104 and the peripheral image identification area 105 represents the edge of the desk on which the product to be checked in is placed. When the user presses the capture button displayed in the message display area 107 in the display example (1) of FIG. 9, the camera 28 captures an image of the product, and the display example (2) of FIG. 9 is displayed.

[0071] 9, the user can press the "take another photo" button displayed in message display area 107 to return to display example (1), allowing the user to take another photo of the product. When the user presses the OK button displayed in message display area 107, information processing terminal 2 transmits the entire image including internal area 104 and peripheral image identification area 105 to inference device 60 as captured image 83. Object identification AI model 43 of inference device 60 identifies the product shown in captured image 83.

[0072] The user interface shown in display examples (1) and (2) in FIG. 9 is characterized in that the size of the internal area 104 represented by the bounding box 106 is a rectangle narrower than the shooting area 103 representing the shooting range. Another characteristic is that the captured image and area information including the coordinates of the bounding box 106 within the image are transmitted to the inference device 60. This bounding box 106 is not drawn in the check-in video, but the coordinates x and y of the upper left vertex of the bounding box 106, as well as the width w and height h of the bounding box 106, are transmitted to the inference device 60 as area information (x, y, w, h).

[0073] 9 shows an example of displaying the check-in determination result. As the check-in determination result, a message is displayed indicating that the check-in was successful because the identified product combination is valid and the photographed image 83 is not fraudulent, and that a benefit (an example of a reward) will be sent to the user whose check-in was successful.

[0074] FIG. 10 is a diagram showing an example of the check-in screen W1 displayed when check-in has failed. In display example (1) of Figure 10, only the drink and burger are displayed out of a set containing multiple items. If the user presses the capture button in this state, an image of only the drink and burger is obtained, as shown in display example (2) of Figure 10. In this case, check-in is determined to have failed.

[0075] If the check-in is determined to be unsuccessful, a message indicating that the check-in was unsuccessful is displayed in the message display area 107, as shown in the display example (3) of Fig. 10. The message display area 107 also displays information including a request to re-photograph the set after checking all the products included in the set to be checked in, and the names of the products that the object identification AI model 43 was unable to identify.

[0076] (Example of functional configuration of reward system) Next, an example of the functional configuration of the reward granting system 10 according to the first embodiment will be described. 11 is a block diagram showing an example of the functional configuration of the reward granting system 10 according to the first embodiment. The reward granting system 10 according to the first embodiment is configured as a client-server system in which the information processing terminal 2 is a client and the inference device 60 is a server.

[0077] First, an example of the functional configuration of the information processing terminal 2 will be described. The information processing terminal 2 includes a photographing area display unit 81, a photographing unit 82, a photographed image 83 stored in the recording device 22 (see FIG. 2), a terminal side communication unit 84, and a result display unit 85.

[0078] The photographing area display unit 81 displays the photographing area 103, the boundary frame 106, etc. on the display device (output device 27) of the information processing terminal 2. The photographing unit 82 generates a photographed image 83 by capturing an image of a product shown in the photographing area 103 using the function of the camera 28 shown in Fig. 2. The photographed image 83 is recorded in the recording device 22 and output to the terminal-side communication unit 84 by a dedicated application started on the information processing terminal 2.

[0079] The terminal-side communication unit 84 transmits the captured image 83 to the inference device 60 via the network N. In addition to the captured image 83, the terminal-side communication unit 84 also transmits area information of an internal area 104 that is set smaller than the captured image 83, and user information of the user operating the information processing terminal 2. The area information of the internal area 104 indicates the position and size of the internal area 104 within the captured image 83, and is represented by (x, y, w, h) shown in FIG. 8. The terminal-side communication unit 84 also receives an object identification judgment result from the inference device 60.

[0080] The result display unit 85 displays the check-in determination result received by the terminal-side communication unit 84. As will be described later, the check-in determination result includes either a successful check-in or a failed check-in. If the check-in is successful, the user can expect to receive a benefit linked to the user information that the user has. On the other hand, if the check-in is unsuccessful, the result includes information about a product that the object identification AI model 43 was unable to identify. Therefore, the user can re-photograph the product based on the check-in determination result displayed by the result display unit 85.

[0081] Next, an example of the functional configuration of the inference device 60 will be described. The inference device 60 includes an inference device side communication unit 91, a captured image 83 recorded in a recording device 64, an image normalization unit 92, an object identification AI model 43, an object identification judgment unit 93, an image classification AI model 53, a score judgment unit 94, and a judgment result table 95.

[0082] The inference device communication unit 91 acquires a captured image 83 of multiple products, transmitted from the terminal communication unit 84 of the information processing terminal 2, and a surrounding image identification area 105, which is set in a part of the captured image 83 and is used to identify images surrounding the products. The surrounding image identification area 105 is an area identified from the captured image 83 by information on the coordinates of one corner of a rectangular boundary line, and a width and height based on the one corner, as shown in FIG. 8, and is acquired as area information. The inference device communication unit 91 also acquires user information transmitted from the terminal communication unit 84 of the information processing terminal 2 via the network N. The captured image 83, the area information, and the user information are stored in the recording device 64 of the inference device 60. The inference device communication unit 91 also transmits an object identification determination result by the object identification determination unit 93 to the terminal communication unit 84 of the information processing terminal 2 via the network N.

[0083] The image normalization unit 92 normalizes the captured image 83 read from the recording device 64. Products are photographed with various aspect ratios depending on the type of information processing terminal 2. The captured image 83 is normalized so that the object identification AI model 43 can process the captured images 83 photographed with various aspect ratios in a common manner. Normalization of the captured image 83 is extremely important in improving the accuracy of product recognition by the object identification AI model 43.

[0084] The image normalization unit 92 calculates the size of the internal region 104 corresponding to the image with the normalized size. This is called the normalization frame. The image normalization unit 92 then determines the resampling magnification in accordance with the ratio between the normalization frame and the size of the captured image 83 to be input. After determining the region to be left after resampling, the image normalization unit 92 actually resamples the captured image 83, thereby enabling normalization of the captured image 83 centered on the internal region 104.

[0085] The object identification AI model 43 is copied from the recording device 34 of the learning device 30 to the recording device 64 of the inference device 60. The object identification AI model 43 receives the normalized captured image 83 as input, sets an object detection area 110 (see FIG. 14 described later) in which the product is detected for each of a plurality of products appearing in the captured image 83, and identifies the product for each object detection area 110.

[0086] The object identification determination unit 93 performs a first determination for authenticating a user based on the position of the object detection area relative to the surrounding image identification area 105. As shown in Fig. 15, which will be described later, the object identification determination unit 93 determines that the first determination performed for authentication has failed if the overlap between the object detection area 110 and the surrounding image identification area 105 is equal to or greater than a predetermined amount (for example, equal to or greater than 10 to 30% of the area of ​​the surrounding image identification area 105).

[0087] Furthermore, the object identification determination unit 93 determines that the first determination performed for authentication is a failure if the overlap between the object detection area 110 and the peripheral image identification area 105 is less than a predetermined amount and the group of identified products does not satisfy the grouping condition. The grouping condition is used to determine whether the group of identified products corresponds to a predetermined combination of multiple different products. Therefore, the object identification determination unit 93 determines that the first determination performed for authentication is a success if the overlap with the peripheral image identification area 105 is less than a predetermined amount and the group of identified products satisfies the grouping condition.

[0088] The object identification judgment result determined by the first judgment is transmitted to the information processing terminal 2 via the inference device side communication unit 91. The object identification judgment result is also recorded in a judgment result table 95. The contents recorded in the judgment result table 95 can be referenced by the administrator of the inference device 60 as needed.

[0089] In this way, the inference device 60 can implement a process that does not allow check-in if the area of ​​the product detected by the object identification AI model 43 deviates from the shooting range at the center of the screen, i.e., if the product image encroaches on the peripheral image identification area 105 and the peripheral image identification area 105 becomes smaller than a certain area (for example, less than 70 to 90% of the area of ​​the peripheral image identification area 105). Therefore, the reward granting system 10 can realize a mechanism that naturally forces the user to take a photo of the peripheral image identification area. Furthermore, in this embodiment, the object identification AI model 43 can identify products without depending on the size or aspect ratio of a specific bounding box 106.

[0090] Image classification AI model 53 is copied from recording device 34 of learning device 30 to recording device 64 of inference device 60. When object identification determination unit 93 determines that the first determination is successful, image classification AI model 53 classifies captured image 83 as a positive example or a negative example and outputs a score.

[0091] The score determination unit 94 performs a second determination for authentication, determining the captured image 83 as a positive example if the score output by the image classification AI model 53 is less than the classification threshold, and determining the captured image 83 as a negative example if the score is equal to or greater than the classification threshold. This second determination is performed to detect check-in fraud using the captured image 83 including not only the internal area 104 shown in FIG. 8 but also the surrounding image identification area 105.

[0092] Note that the scores output by the image classification AI model 53 have a range of values, and using only one classification threshold may result in a captured image 83 that should be classified as a positive example being determined as a negative example, or conversely, a captured image 83 that should be classified as a negative example being determined as a positive example. For this reason, the score determination unit 94 classifies the captured image 83 by dividing the classification threshold into a first classification threshold and a second classification threshold that is greater than the first classification threshold. As shown in FIG. 17 (described later), the score determination unit 94 determines the captured image 83 as "Pass," indicating a positive example, if the score is less than the first classification threshold, and as "Fail," indicating a negative example, if the score is equal to or greater than the second classification threshold. Captured images 83 determined as "Fail" are subject to audit by an administrator. Furthermore, if the score is equal to or greater than the first classification threshold and less than the second classification threshold, the score determination unit 94 determines the positive example or negative example as "Borderline," which cannot be classified. Captured images 83 determined as "Borderline" are subject to review by an administrator. The score output by the image classification AI model 53 is recorded in the judgment result table 95 as the judgment result of the second judgment by the score judgment unit 94, and the object identification judgment unit 93 can refer to the judgment result of the second judgment.

[0093] In the reward granting system 10, a reward is granted to a user whose first judgment is judged to be successful and whose second judgment is judged to be "Pass." For example, a reward is granted to a user whose captured image 83 is judged to be a positive example by the score judgment unit 94. On the other hand, if the score judgment unit 94 judges the captured image 83 to be a negative example, user authentication fails. A user who sends a captured image 83 that is judged to be a negative example is likely to have performed fraudulent operations. For users who perform fraudulent operations, even if a reward was once granted, the administrator may cancel the reward or delete the account.

[0094] <Example of processing by an information processing terminal and an inference device> In the check-in service provided to users by the reward granting system 10, the inference device 60 receives, as a check-in token, an image captured by an application and coordinate information of a bounding box 106 of the image from the information processing terminal 2. Note that the reward granting system 10 preferably does not receive images from any client, but rather receives only images captured from a specific application or a specific web page. Therefore, secure session management is required, for example, so that only specific users can transmit the captured image 83 used to determine check-in.

[0095] Here, the processing on the side of the inference device 60 will be described. Here, the check-in service performed by the information processing terminal 2 and the inference device 60 will be described with reference to an example of a screen displayed on the information processing terminal 2 shown in FIG. FIG. 12 is a sequence diagram showing an example of processing by the information processing terminal 2 and the inference device 60.

[0096] A user operating the information processing terminal 2 starts an application and starts the camera 28 via the application. When the camera 28 starts, the output device 27 displays the internal area 104, the photographed area 103 including the surrounding image identification area 105, etc., as shown in the display example (1) of Fig. 9 (S21).

[0097] When the user photographs a set (multiple products) at once (S22), a photographed image 83 showing the multiple products is displayed, as shown in display example (2) of FIG. 9. Also, a message is displayed in message display area 107 to confirm whether the user wants to send the photographed image to the server (inference device 60) and check it in. When the user presses the OK button, terminal-side communication unit 84 transmits photographed image 83, area information, and user information to inference device 60 (S23).

[0098] The inference device-side communication unit 91 of the inference device 60 receives the captured image 83, area information, and user information from the information processing terminal 2 (S31). The inference device-side communication unit 91 records the captured image 83, area information, and user information acquired from the information processing terminal 2 in the recording device 64 (S32).

[0099] Next, an object identification inference process is performed (S33), and the object identification result is output to the object identification determination unit 93. An example of the object identification inference process in step S33 will now be described. FIG. 13 is a flowchart illustrating an example of an object identification inference process.

[0100] First, the image normalization unit 92 acquires the photographed image 83 read from the recording device 64, and normalizes the photographed image 83 (S41).

[0101] Next, the object identification AI model 43 detects the internal region 104 from the normalized photographed image 83 (S42). Next, the object identification AI model 43 cuts out an object detection region of the product shown in the normalized photographed image 83 and performs object recognition (assigning a product name) (S43), and proceeds to step S34.

[0102] FIG. 14 is a diagram showing an example of an object identification result. In the object identification inference process shown in step S33 of Fig. 12 and Fig. 13, products are individually identified based on a captured image 83 that shows multiple products. For example, in the captured images (1) to (4) of Fig. 14, even though the cups, burgers, and fries are randomly arranged, the product names of the identified products are labeled in the object detection areas 110 indicated by the bounding boxes detected for each product. Note that, as shown in the captured images (3) and (4), even though the cups and burgers are partially hidden by the fries, the products are correctly identified.

[0103] Returning to the explanation of FIG. In the object identification inference process, the object identification AI model 43 detects a product based on the captured image 83. At this time, the object identification AI model 43 outputs the label (identifier string) of the identified product, the score of the identified product, and detection area information (x, y, w, h) representing the object detection area 110 to the object identification determination unit 93 as the object identification result.

[0104] The object identification determination unit 93 performs an object identification determination process based on the object identification result (S34). An example of the object identification determination process in step S34 will now be described. 15 is a flowchart showing an example of an object identification determination process. The object identification determination process is an example of a first determination performed for authenticating a user.

[0105] First, the object identification determination unit 93 compares the peripheral image identification region 105 acquired by the inference apparatus communication unit 91 in step S31 with the object detection region 110 identified from the detection region information (S51). Next, the object identification determination unit 93 determines whether the overlap of the object detection region 110 with the peripheral image identification region 105 is equal to or greater than a predetermined amount (S52). The overlap of the object detection region 110 with the peripheral image identification region 105 is determined, for example, based on the ratio of the area of ​​the object detection region 110 that extends into the peripheral image identification region 105 to the area of ​​the peripheral image identification region 105. For example, if the overlap area of ​​the object detection region 110 with the peripheral image identification region 105 exceeds 10 to 30% of the area of ​​the peripheral image identification region 105, it is determined that the overlap is equal to or greater than a predetermined amount. Note that the overlap of the object detection region 110 with the peripheral image identification region 105 may be determined using coordinate calculation, a collision determination check, or the like.

[0106] In step S52, if the object identification determination unit 93 determines that the overlap of the object detection area 110 with the surrounding image identification area 105 is equal to or greater than a predetermined amount (YES in S52), the object identification determination unit 93 determines that the check-in is a "failure." Then, the object identification determination unit 93 outputs the object identification determination result that is a "failure" (S53), and the process returns to step S35 in FIG. 12.

[0107] If the object identification determination unit 93 determines in step S52 that the overlap is less than a predetermined amount (NO in S52), it determines whether the set of products identified by the object identification AI model 43 satisfies the set condition (S54). Specifically, the object identification determination unit 93 confirms by pattern matching of the identifier string that the combination of product names included in the set menu is included in the label output from the object identification AI model 43.

[0108] In step S54, if the object identification determination unit 93 determines that the collection of identified products satisfies the collection condition (YES in S54), it determines that the check-in is "successful", that is, that the first determination is successful. Then, the object identification determination unit 93 outputs the object identification determination result that is "successful" (S55), and returns to step S35 in FIG. 12.

[0109] On the other hand, if the object identification determination unit 93 determines in step S54 that the set of identified products does not satisfy the set condition (NO in S54), it determines that the check-in is a "failure," i.e., that the first determination is a failure. Then, the object identification determination unit 93 outputs the object identification determination result including the fact that the check-in is determined to be a "failure" and the product names that do not correspond to the combination (S56), and returns to step S35 in Fig. 12. By outputting the product names that do not correspond to the predetermined combination among the products identified by the object identification AI model 43, a message about the products that do not correspond to the predetermined combination is displayed on the information processing terminal 2, as shown in display example (3) in Fig. 10.

[0110] Here, an example of an object identification result by the object identification AI model 43 will be described. FIG. 16 is a diagram showing an example of an object identification result and a check-in result. In the object identification result (1) of the captured image 83 shown in Figure 16, the state of each product detected by the object identification AI model 43 in the captured image 83 is shown by the object detection area 110, which is a bounding box of a rectangular frame indicated by a dashed line. Each object detection area 110 is labeled with the product name identified by the object identification AI model 43. Because the object detection area 110 of each product detected by the object identification AI model 43 is inside the internal area 104, the check-in is determined to be "successful."

[0111] 16, the object identification result (2) for the captured image 83 shows the detection by the object identification AI model 43 for each product shown in the captured image 83 in the object detection area 110, which is a rectangular frame indicated by a dashed line. However, since the overlap between the surrounding image identification area 105 and the object detection area 110 labeled Cup and Potato is greater than or equal to a predetermined amount, the check-in is determined to be a "failure."

[0112] Returning to the explanation of FIG. The inference device communication unit 91 transmits the object identification judgment result to the information processing terminal 2 whether the check-in judgment is successful or unsuccessful (S35). In addition, the object identification judgment result, which is the result of the first judgment, is recorded in a judgment result table 95, as shown in FIG.

[0113] The terminal-side communication unit 84 of the information processing terminal 2 receives the object identification judgment result from the inference device 60 (S24). The output device 27 of the information processing terminal 2 displays the judgment result based on the object identification judgment result (S25), and the processing of the information processing terminal 2 ends.

[0114] On the other hand, in inference device 60, if the check-in is determined to be "successful" in step S34, that is, if the first determination is determined to be successful, the image classification inference process continues (S36). Here, an example of the image classification inference process of step S36 will be described. FIG. 17 is a flowchart illustrating an example of an image classification inference process.

[0115] First, the image classification AI model 53 classifies the captured image 83 read from the recording device 64 and outputs a score (S61). Next, the score determination unit 94 performs a second determination for authentication, comparing the score with a classification threshold to determine whether the captured image 83 is a positive example or a negative example.

[0116] Specifically, the score determination unit 94 determines whether the score is less than the first classification threshold (S62). If the score is less than the first classification threshold (YES in S62), the score determination unit 94 writes "Pass" in the determination result table 95 and ends the process. A captured image 83 being "Pass" means that it is a legitimate image.

[0117] If the score is equal to or greater than the first classification threshold (NO in S62), the score determination unit 94 determines whether the score is equal to or greater than a second classification threshold that is greater than the first classification threshold (S64). If the score is equal to or greater than the second classification threshold (YES in S64), the score determination unit 94 writes "Fail" in the determination result table 95 and ends the process. If the captured image 83 is "Fail," it means that it is an invalid image.

[0118] If the score is not equal to or greater than the second classification threshold (NO in S64), the score is equal to or greater than the first classification threshold but less than the second classification threshold. In this case, the score determination unit 94 writes "Borderline" to the determination result table 95 and ends the process. If the captured image 83 is "Borderline," it means that it is not known whether it is a legitimate image or an unauthorized image.

[0119] <Example of judgment result table configuration> 18 is a diagram showing an example of the configuration of the determination result table 95. The determination result table 95 manages user information, the captured image 83, and the determination results of each AI model. The determination result table 95 has the following items: user ID, image reception date and time, photographed image, object identification result, object identification determination result, image classification result, future processing, and authentication result.

[0120] The user ID item stores the user ID of the user who sent the photographed image 83. If one user sends an image multiple times, a record is created for each transmission. The image reception date and time item stores the date and time (year, month, day, and time) when the inference device 60 received the captured image 83 transmitted from the information processing terminal 2. The captured image item stores the file of the captured image 83 received from the information processing terminal 2. Here, an example is shown in which an image file in JPEG format is stored, but any extension is acceptable as long as it is an image file. The captured image item may also store location information (such as a path) that indicates the location of the file of the captured image 83.

[0121] The object identification result item stores the object identification result obtained by the object identification AI model 43 identifying the product shown in the captured image 83. Here, the name of the product included in the set menu is stored as the object identification result. The object identification judgment result field stores "success" or "failure" as the object identification judgment result.

[0122] The image classification result field stores the image classification result determined by the score determination unit 94 based on the score with which the image classification AI model 53 classifies the captured image 83 as a positive example or a negative example. As described above, the fact that the image classification AI model 53 has classified the captured image 83 as a positive example is represented by "Pass," and the fact that it has classified the captured image 83 as a negative example is represented by "Fail." Note that the image classification AI model 53 represents the image classification result as a score, and if neither the score for the positive example nor the negative example is significantly high and manual verification is required, the image classification result is represented by "Borderline." Furthermore, if the object identification determination result is "Fail," the image classification inference process is not performed, and the image classification result field remains blank.

[0123] The future processing item stores future processing to be performed based on the object identification result and the image classification result. The authentication result item stores the final authentication result. The final authentication result may be registered by the administrator because even if the first judgment is successful, the second judgment may fail.

[0124] When the object identification judgment result is "Success" and the image classification result is "Pass", as in the case of user1, the final user authentication is judged to be "Success", and a benefit (an example of a reward) is granted to the user.

[0125] If the object identification judgment result for the first-sent captured image 83 is "failed," as in the case of user2, the final user authentication is judged as "failed," and no reward is given to the user. However, if the object identification judgment result for the second-sent captured image 83 is "successful," an image classification result is obtained. If the image classification result is "pass," the final user authentication is judged as "successful," and a reward is given to the user.

[0126] In the case of user3, even if the object identification judgment result is "Success," if the image classification result is "Fail," there is a high possibility that the captured image 83 is fraudulent. For this reason, the administrator of the inference device 60 audits the captured image 83. If it is confirmed that the captured image 83 was acquired fraudulently, the privilege for user3 is revoked, for example. That is, even if the object identification judgment result is "Success" and the image classification result is "Pass," if the final user authentication is determined to be "Fail," the privilege is not granted to the user. In this embodiment, because the image classification AI model 53 classifies fraudulent captured images 83, a user attempting fraudulent operations (cheating) must necessarily repeatedly try and error the fraudulent operations. As a result, a history of fraudulent operations is accumulated in the judgment result table 95, making it easy for the administrator to take measures such as revoking privileges for users attempting fraud or suspending their accounts.

[0127] In the case of User 4, even if the object identification judgment result is "success," if the image classification result is "Borderline," the image classification AI model 53 was unable to correctly classify the captured image 83, and the captured image 83 is then checked by the administrator. As a result, the final user authentication is either "success" or "failure," and the result checked by the administrator is stored in the authentication result field.

[0128] The reward granting system 10 according to the first embodiment described above uses product wrapping paper, which contains a lot of noise due to folding and dirt, as a check-in point, and is able to accurately identify the combination of products, i.e., the products included in a "set menu." Therefore, the object recognition AI model 43 can simultaneously identify multiple products to be checked in from the products depicted in any photographed image 83. Furthermore, the image classification AI model 53 classifies the photographed image 83 using the images depicted in the peripheral image recognition area 105 to determine whether the photographed image 83 has been fraudulently created or tampered with, and outputs the classification result as a score. The score determination unit 94 can determine the score and classify the image. Therefore, the inference device 60 uses product wrapping paper, which contains a lot of noise due to folding and dirt, as a check-in point, and is able to simultaneously identify the combination of products, i.e., the "set menu," and implement fraud prevention measures, thereby preventing fraud.

[0129] Furthermore, by combining the object identification AI model 43 and the image classification AI model 53, the reward granting system 10 can quickly (within 1 second of processing time) and accurately identify set menus containing a variety of products while being robust against fraud. Specifically, it is possible to detect a collection of predetermined products from a captured image 83 containing noise and occlusions captured using the camera 28 mounted on the information processing terminal 2, and further to check that the captured image has not been intentionally tampered with or altered.

[0130] Furthermore, a check-in screen W1 that the user uses to perform a check-in operation is displayed on the output device 27 of the information processing terminal 2. A rectangular boundary frame 106 that is narrower than the actual shooting area 103 is displayed in the center of the check-in screen W1. This makes it easier for the user to perform operations so that the product to be checked in is captured within the boundary frame 106.

[0131] Furthermore, if the object detection area of ​​the product detected by the object identification AI model 43 protrudes from the internal area 104 specified by the area information, the object identification determination unit 93 determines the object identification determination result as "failure" and does not permit check-in. If the object identification determination result is determined to be "failure," a message urging the user to take a photo so that the product is visible inside the boundary frame 106 is displayed on the check-in screen W1. This allows the user to retake the photo and attempt check-in again.

[0132] Furthermore, to determine whether the captured image 83 is a fraudulent image, an image of the surrounding image identification area 105 is particularly required. By naturally forcing the user to capture a photograph so that the product is inside the bounding box 106, an image of the surrounding image identification area 105 large enough to classify the image as fraudulent is also captured. As a result, the classification accuracy of the image classification AI model 53 is improved, and the tamper resistance of the reward granting system 10 can be enhanced.

[0133] Furthermore, for example, an in-game reward can be given to a user whose object identification judgment result is determined to be successful and whose photographed image 83 is classified as legitimate. This makes it possible to promote sales by motivating users to purchase the menu item that is the target of their check-in.

[0134] [Second embodiment] Next, a reward system according to a second embodiment of the present invention will be described. In the client-server system configured with the reward system 10 according to the first embodiment, the approach of placing an AI model on the server side is merely one embodiment. For example, by using a small AI model for check-in purposes, services can be flexibly designed according to the scale and budget of the project. It is also possible to install a small AI model on the information processing terminal 2 and perform check-ins without network communication.

[0135] Here, a reward granting system 10A according to the second embodiment will be described with reference to Fig. 19. Fig. 19 is a block diagram showing an example of the overall configuration of the reward granting system 10A according to the second embodiment.

[0136] The reward granting system 10A includes an information processing terminal 2A and an inference device 60A. The information processing terminal 2A includes a terminal object identification AI model 45, an image normalization unit 86, and an object identification determination unit 87 in addition to the functional units of the information processing terminal 2 described with reference to FIG. The object identification AI model for terminal 45 is an optimized AI model converted from the object identification AI model 43 by the object identification AI model conversion unit 44 shown in Fig. 4. By preparing the object identification AI model for terminal 45 in advance, the information processing terminal 2A can cause the object identification AI model for terminal 45 to perform operations equivalent to those of the object identification AI model 43.

[0137] The image normalization unit 86 has the same function as the image normalization unit 92 shown in FIG. The object identification and determination unit 87 has the same function as the object identification and determination unit 93 shown in FIG. Therefore, in the information processing terminal 2A according to the second embodiment, the image normalization unit 86 normalizes the captured image 83, the object identification AI model 45 for the terminal identifies the product appearing in the normalized captured image 83, and the object identification determination unit 87 can independently perform the process of determining check-in based on the object identification result.

[0138] The captured image 83 is transmitted to the inference device 60A via the terminal-side communication unit 84. The inference device-side communication unit 91 records the captured image 83 in the recording device 64. The image classification AI model 53 classifies the captured image 83 as a positive example or a negative example and outputs a score, and the score determination unit 94 records the image classification result with the determined score in a determination result table 95.

[0139] In the reward granting system 10A according to the second embodiment described above, the terminal object identification AI model 45 provided in the information processing terminal 2A identifies the product shown in the captured image 83, and a determination is made by the object identification determination unit 87. Therefore, if the purpose is only check-in, the processing can be completed by the information processing terminal 2A, thereby reducing the operational load on the inference device 60A.

[0140] [Third embodiment] Next, a reward granting system according to a third embodiment of the present invention will be described. The reward granting system according to the third embodiment is composed of only an information processing terminal 2B.

[0141] Here, a reward granting system 10B according to the third embodiment will be described with reference to Fig. 20. Fig. 20 is a block diagram showing an example of the overall configuration of the reward granting system 10B according to the third embodiment.

[0142] The reward granting system 10B includes an information processing terminal 2B. In addition to the functional units of the information processing terminal 2A described with reference to Fig. 19, the information processing terminal 2B includes a terminal image classification AI model 55, a score determination unit 88, and a determination result table 89.

[0143] Terminal image classification AI model 55 is an optimized AI model converted from image classification AI model 53 by image classification AI model conversion unit 54 shown in Fig. 4. By preparing terminal image classification AI model 55 in advance, information processing terminal 2A can cause terminal image classification AI model 55 to perform operations equivalent to image classification AI model 53.

[0144] As with the reward granting system 10A according to the second embodiment, the information processing terminal 2B can independently identify the product from the photographed image 83 and determine whether or not it has been checked in. Furthermore, in the information processing terminal 2B, the terminal image classification AI model 55 classifies the captured image 83 into a positive example or a negative example and outputs a score, and the score determination unit 88 records the image classification result determined as a score in a determination result table 89. Note that the determination result table 89 may be encrypted so that the user cannot read its contents. Also, the information in the determination result table 89 may be transmitted to a server managed by an administrator.

[0145] In the reward granting system 10B according to the third embodiment described above, the terminal object identification AI model 45 provided in the information processing terminal 2B identifies a product shown in the captured image 83, and a determination is made by the object identification determination unit 87. Furthermore, the terminal image classification AI model 55 provided in the information processing terminal 2B determines a score based on the score obtained when the captured image 83 is classified as a positive example or a negative example. Therefore, even in an offline environment, the information processing terminal 2B can identify a product from the captured image 83 and perform the process of classifying the captured image 83 as a positive example or a negative example.

[0146] The AI ​​models according to the second and third embodiments can be run on either the server or the client. If the check-in measure is long-term or permanent, it can be run on the client information processing terminal 2, and if it is short-term, it can be run on the server inference device 60, thereby achieving low costs.

[0147] [Variations] In addition, by using WebGPU EP (Execution Provider), it is possible to load the AI ​​model directly into the web browser and perform inference without going through an application. Therefore, the reward system does not necessarily have an architecture that relies on a client / server system.

[0148] Furthermore, the AI ​​model capable of realizing each of the above-described embodiments does not depend on a specific AI model (such as the above-described YOLO or ResNet). AI models constructed using various technologies may be used as the object identification AI model 43 and the image classification AI model 53.

[0149] It should be noted that the present invention is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the gist of the present invention as set forth in the claims. For example, the above-described embodiments have described the configuration of the device and system in detail and specifically in order to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of the embodiments described here with the configuration of other embodiments, and it is also possible to add the configuration of one embodiment to the configuration of another embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0150] 2...information processing terminal, 10...reward granting system, 30...learning device, 41...learning data for object identification, 42...object identification learning unit, 43...object identification AI model, 44...object identification AI model conversion unit, 45...object identification AI model for terminal, 51...learning data for image classification, 52...image classification learning unit, 53...image classification AI model, 54...image classification AI model conversion unit, 55...image classification AI model for terminal, 60...inference device, 81... Photographing area display section, 82...photographing section, 83...photographed image, 84...terminal side communication section, 85...result display section, 91...inference device side communication section, 92...image normalization section, 93...object identification judgment section, 94...score judgment section, 95...judgment result table, 101...photographable area, 102...display area, 103...photographing area, 104...internal area, 105...surrounding image identification area, 106...boundary frame, 107...message display area, W1...check-in screen

Claims

1. a step of acquiring a photographed image in which a plurality of objects are photographed, and a peripheral image identification region set in a part of the photographed image for identifying peripheral images of the objects; a step of setting an object detection area in which the object identification model detects each of the plurality of objects appearing in the captured image, and identifying the object for each of the object detection areas; determining that user authentication has failed if the overlap between the object detection area and the peripheral image identification area is equal to or greater than a predetermined amount, determining that a first determination made for the authentication has failed if the overlap between the object detection area and the peripheral image identification area is less than the predetermined amount and the collection of identified objects does not satisfy a collection condition, and determining that the first determination has succeeded if the overlap between the object detection area and the peripheral image identification area is less than the predetermined amount and the collection of identified objects satisfies the collection condition. An inference program to be executed by a computer.

2. The set condition is that the set of identified objects corresponds to a predetermined combination of a plurality of different predetermined objects. The inference program according to claim 1 .

3. Among the objects identified by the object identification model, information on the objects that do not correspond to a predetermined combination is output. The inference program according to claim 2 .

4. a step of outputting a score indicating that the image classification model has classified the captured image as a positive example or a negative example when the first determination is determined to be successful; and a step of performing a second determination for the authentication, determining the captured image as the positive example if the score is less than a classification threshold, and determining the captured image as the negative example if the score is equal to or greater than the classification threshold. The inference program according to claim 1 .

5. A display unit of an information processing terminal that captures images of the plurality of objects displays a circular boundary line indicating the peripheral image identification area. The inference program according to claim 1 .

6. The boundary line is rectangular, the area of ​​the peripheral image identification area is smaller than the area of ​​the photographing area displayed on the display unit; The surrounding image identification area is identified from the captured image by information on the coordinates of one corner of the rectangular boundary line, and a width and height based on the one corner. The inference program according to claim 5 .

7. The display unit displays the peripheral image identification area with a lower brightness than the area inside the boundary line. The inference program according to claim 6.

8. acquiring a photographed image of a plurality of objects and a peripheral image identification region set in a part of the photographed image for identifying peripheral images of the objects; a step of setting an object detection area in which the object identification model detects each of the plurality of objects appearing in the captured image, and identifying the object for each of the object detection areas; determining that user authentication has failed if the overlap between the object detection area and the surrounding image identification area is equal to or greater than a predetermined amount, determining that a first determination made for the authentication has failed if the overlap between the object detection area and the surrounding image identification area is less than the predetermined amount and the collection of identified objects does not satisfy a collection condition, and determining that the first determination has succeeded if the overlap between the object detection area and the surrounding image identification area is less than the predetermined amount and the collection of identified objects satisfies the collection condition. Reasoning method.

9. a communication unit that acquires a captured image of a plurality of objects and a peripheral image identification area that is set in a part of the captured image and that is used to identify peripheral images of the objects; an object identification model that sets an object detection area in which each of the plurality of objects appearing in the captured image is detected and identifies the object for each of the object detection areas; a determination unit that determines user authentication to be a failure when the overlap between the object detection region and the peripheral image identification region is equal to or greater than a predetermined amount, determines a first determination performed for the authentication to be a failure when the overlap between the object detection region and the peripheral image identification region is less than the predetermined amount and the collection of identified objects does not satisfy a collection condition, and determines the first determination to be a success when the overlap between the object detection region and the peripheral image identification region is less than the predetermined amount and the collection of identified objects satisfies the collection condition. Reasoning device.

Images