Information processing system

Abstract

To provide a technique for enhancing effectiveness for means for identifying a free gift which is stored in an article acquisition game device.SOLUTION: An information processing system comprises: an advertisement image acquisition part for acquiring, an article ID and an advertisement image of an article identified by the article ID, from a database which stores information on the article which is an acquisition object in an article acquisition game device; a model learning part for generating a determination model for identifying the article ID from a picked up image of the article, on the basis of first teacher data in which the article ID is an objective variable and the advertisement image is an explanatory variable; and an article identification part for identifying the article ID of the article which is stored, on the basis of the picked up image of the article stored in the article acquisition game device, and the determination model.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an item acquisition game for acquiring items such as prizes.

Background Art

[0002] Many game centers have a crane game device (item acquisition game device) installed. A stage (game field) is provided inside the housing of the crane game device, and prizes (items) such as stuffed toys and snacks are placed on it. The player operates the crane by using the horizontal and vertical buttons. If the player can grab the prize with the crane and move the prize to the drop hole, the player can acquire the prize.

[0003] Stores such as game centers need to know which prizes are accommodated in which crane game device. Therefore, when an operator of the store puts a prize into the crane game device, the operator operates the store terminal to register the prize that has been put in. This registration by manual input takes time.

[0004] The inventors of the present invention considered using AI (Artificial Intelligence) technology as a method for automating prize registration. The prize that has been put in is photographed by a camera installed in the crane game device, and the image is analyzed to identify the prize ID. If the prize ID can be automatically identified, manual input is no longer necessary.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] For AI-based prize identification, it is assumed that a judgment model obtained through supervised machine learning will be used. As training data for machine learning, a large number of sample images must be prepared for each prize. Furthermore, there are thousands of different prizes offered for crane game machines. Therefore, the preparation burden for training the judgment model when using AI technology to identify prizes is extremely large.

[0007] This invention was completed in view of the above background, and its main purpose is to provide a technology that enhances the effectiveness of the means for identifying prizes contained in a crane game machine. [Means for solving the problem]

[0008] An information processing system in one aspect of the present invention includes: an advertising image acquisition unit that acquires an item ID and an advertising image of an item identified by the item ID from a database that stores information on items to be acquired in an item acquisition game device; a model learning unit that generates a determination model that identifies an item ID from a photographed image of an item based on first training data in which the item ID is the target variable and the advertising image is the explanatory variable; and an item identification unit that identifies an item ID of an item stored inside the item acquisition game device based on a photographed image of the item and the determination model.

[0009] An information processing device in one aspect of the present invention includes: an advertising image acquisition unit that acquires an item ID and an advertising image of an item identified by the item ID from a database that stores information on items to be acquired in an item acquisition game device; and a model learning unit that generates a determination model that identifies an item ID from a photographed image of an item, based on first training data in which the item ID is the target variable and the advertising image is the explanatory variable. [Effects of the Invention]

[0010] According to the present invention, it becomes easier to effectively utilize the means for identifying the prizes contained in the item acquisition game device. [Brief explanation of the drawing]

[0011] [Figure 1] It is a hardware configuration diagram of a game system. [Figure 2] It is a diagram schematically showing the overall configuration of a crane game device. [Figure 3] It is a functional block diagram of a store support server. [Figure 4] It is a functional block diagram of a crane game device. [Figure 5] It is a diagram showing an example of a promotional image. [Figure 6] It is a flowchart showing the processing process of the store support server in the learning phase. [Figure 7] It is a schematic diagram showing a first method of extracting a partial image from an entire image. [Figure 8] It is a schematic diagram showing a second method of extracting a partial image from an entire image. [Figure 9] It is a flowchart showing the processing process of the store support server in the operation phase. [Figure 10] It is a data configuration diagram of a store data storage unit. [Figure 11] It is a diagram showing a first example of a derived image. [Figure 12] It is a diagram showing a second example of a derived image. [Figure 13] It is a diagram showing a third example of a derived image. [Figure 14] It is a diagram showing an example of a partial image including non-target objects. [Figure 15] It is a diagram showing an example of a partial image of a target prize. [Figure 16] It is a diagram showing the sequence in Variation 3. [Figure 17] It is a diagram showing the sequence in Variation 7.

Embodiments for Carrying Out the Invention

[0012] In this embodiment, AI technology is used to identify the prizes stored in the crane game device. The processing operations are divided into a learning phase and an operation phase.

[0013] In the learning phase, machine learning is performed using sample images of the prizes as teacher data to generate a determination model. In this embodiment, focusing on the fact that catalog data is always prepared for all prizes, the promotional images of the catalog data are used as the sample images of each prize. By using the catalog data, it is not necessary to take pictures, so the labor of preparing the teacher data is reduced. In addition, a determination model learned for many prizes can determine various prizes.

[0014] In the operation phase, an arbitrary prize is put into the crane game device. The prize put in is photographed by a camera provided in the crane game device. By using the photographed image and the determination model, the prize can be automatically identified. Also, the registration process regarding which prize is in which crane game device can be automated.

[0015] Figure 1 is a hardware configuration diagram of the game system. In the game system 100, the user terminals 110a ··· 110m (hereinafter, collectively referred to as "user terminals 110" when mentioned together or not particularly distinguished), the in-store server 107 provided on the store side, and the store terminal 109 are connected to the store support server 102 via the Internet 106. A plurality of crane game devices 104a, 104b ··· 104n (hereinafter, collectively referred to as "crane game devices 104" when mentioned together or not particularly distinguished) are connected to the store support server 102 via the in-store server 107 and the Internet 106.

[0016] The crane game machine 104 is installed in amusement facilities such as amusement parks and game centers. A store is described below as an example of an amusement facility. The store has a store LAN (Local Area Network) that connects the crane game machine 104 to the store server 107. The LAN can be either wireless or wired. The store server 107 primarily acts as a relay for data communication via the internet 106. The store server 107 may also have data management functions for the store. The store terminal 109 can also connect to the store LAN. The store terminal 109 may be a smartphone, tablet, or laptop PC. The crane game machine 104 is identified by a game machine ID. The crane game machine 104 may also connect directly to the internet 106 without going through the store server 107.

[0017] In this embodiment, the user terminal 110 (communication terminal) is assumed to be a smartphone. The user terminal 110 may also be a tablet terminal or a laptop PC. The user terminal 110 and the internet 106 are connected wirelessly, but they may also be connected via a wired connection.

[0018] The store support server 102 provides various services to stores and players. In this embodiment, the store support server 102 generates a judgment model. The store support server 102 manages information such as "which prize is in which crane game machine 104 in which store." The store support server 102 is a public server and accepts access from store terminals 109 and the like. It also accepts access from user terminals 110 regarding services for players. This allows players to obtain useful information about the crane game machine 104.

[0019] The prize database 101 is connected to the internet 106. The prize database 101 holds catalog data for promoting the sale of prizes and provides this data upon request from store terminals 109 and other devices. The store support server 102 is connected to the prize database 101 via network 103. Network 103 can be a LAN or the internet. The store support server 102 retrieves catalog data from the prize database 101 and uses the promotional images contained within it as sample images for training data.

[0020] Figure 2 is a schematic diagram showing the overall configuration of the crane game machine 104. The crane game machine 104 comprises a rectangular base 112 and a box-shaped prize storage section 114 provided on the base 112. The prize storage section 114 stores prizes placed by the store operator. Here, "storage" refers to placing the prizes in a specific or arbitrary position. The state in which the prizes are moved by the crane 118 and remain inside the prize storage section 114 is also considered "storage". A play space S is formed inside the prize storage section 114, and a prize placement platform 116 (game field) is provided. Prizes P, such as stuffed animals and miscellaneous goods, are placed on the prize placement platform 116. A crane 118 is provided above the prize placement platform 116. The crane 118 can move forward, backward, left, right, up, and down within the play space S, and grasps / releases the prizes P.

[0021] The prize storage section 114 has transparent glass on its front and left and right sides, and the front has an openable door. This is to allow for visibility of the prizes P from the outside. A camera 122 is installed on the ceiling of the crane game machine 104. The camera 122 photographs the prize display stand 116 from above. Another camera 122 is installed on the back to photograph the entire play space from an oblique angle above. Only one camera 122 may be installed, or both cameras 122 may be installed. The cameras 122 may be attached to an arm or the like. Camera 122 is an example of a means of taking pictures.

[0022] LEDs (Light Emitting Diodes) are installed on the vertical columns 150 of the crane game machine 104, and they light up as part of the visual effects during gameplay. A door 124 is provided on the front of the prize storage section 114, allowing the operator to place the prizes P inside the prize storage section 114.

[0023] The prize display stand 116 is divided into a first area 126 and a second area 128. Each divided area is made up of removable panels. By removing either panel, a drop-off opening 130 for dropping the prize P can be formed. When the prize P is placed in the first area 126 and the drop-off opening 130 is formed in the second area 128, the player can acquire the prize P if they can move the prize P from the first area 126 to the second area 128 (drop-off opening 130).

[0024] A prize stock space 132 is formed inside the base 112 to accommodate the prizes P that fall from the drop-off opening 130. A prize retrieval opening 134 is formed on the front of the base 112 for retrieving the prizes P that have fallen into the prize stock space 132. The form of the drop-off opening 130 is not limited to this example.

[0025] A control panel 136 is provided on the front side of the base 112. The control panel 136 is equipped with a coin slot 138, an IC (Integrated Circuit) card reader 140, an operation unit 142, and a setting display unit 144. To start the game, the player either inserts a coin into the coin slot 138 or touches an IC card loaded with electronic money to the IC card reader 140. In the latter case, payment processing using electronic money is performed, but since this is a publicly known technology, the details will be omitted.

[0026] The control unit 142 has control buttons 142a and 142b for the player to move the crane 118. The control unit 142 functions as a "crane control unit" that receives input signals based on the player's operations. Control button 142a is a button that moves the crane 118 in the left-right direction (X direction), and control button 142b is a button that moves the crane 118 in the forward-backward direction (Y direction). As a variation, the crane 118 may be moved forward, backward, left, and right using a joystick.

[0027] A touch panel is installed on the settings display unit 144. The settings display unit 144 accepts input of game setting information from the operator and displays information related to the game, such as the operation method of the control unit 142 and game results. The crane game device 104 also includes a speaker (not shown) and external connection terminals.

[0028] The crane 118 has an arm 146 capable of gripping and releasing the prize P. The crane 118 includes a motor that drives the arm 146 to open and close. By opening and closing the arm 146, the prize P is gripped and released.

[0029] The crane 118 is movable along a guide rail (not shown) installed on the top of the prize storage section 114 and is driven by the crane drive unit 148. The crane drive unit 148 includes a movement mechanism that drives the crane 118 in the horizontal (X direction) and vertical (Y direction) directions, and a lifting mechanism that drives it in the vertical (Z direction) direction. The movement mechanism includes an X-direction motor and a Y-direction motor. The lifting mechanism includes a Z-direction motor. The crane drive unit 148 can move the crane 118 to any position in the play space S. Since such a drive mechanism is publicly known, a detailed explanation is omitted. In addition, although not shown, the crane game device 104 has a control unit that receives operation instructions from the operation unit 142 and controls the crane 118, camera 122, LED, setting display unit 144, movement mechanism, lifting mechanism, etc.

[0030] Figure 3 is a functional block diagram of the store support server 102. Each component of the store support server 102 is realized by hardware including arithmetic units such as a CPU (Central Processing Unit) and various coprocessors, storage devices such as memory and storage, and wired or wireless communication lines connecting them, as well as software stored in the storage devices that supplies processing instructions to the arithmetic units. The computer program may consist of device drivers, an operating system, various application programs located at a higher layer, and libraries that provide common functions to these programs. The blocks described below represent functional units, not hardware units.

[0031] The store support server 102 includes a data processing unit 162, a data storage unit 164, and a communication unit 160. The communication unit 160 is responsible for communication processing with the in-store server 107, the crane game machine 104, the store terminal 109, and the user terminal 110. The data storage unit 164 stores various types of data. The data processing unit 162 performs various processes based on the data received by the communication unit 160 and the data stored in the data storage unit 164. The data processing unit 162 also functions as an interface between the communication unit 160 and the data storage unit 164.

[0032] The communication unit 160 includes a transmitting unit 166 that transmits various types of data and a receiving unit 168 that receives various types of data.

[0033] The data processing unit 162 includes an advertising image acquisition unit 300, a model learning unit 302, a data processing unit 308, a data acquisition unit 312, and a data provision unit 314.

[0034] The promotional image acquisition unit 300 acquires promotional images associated with prize IDs from the prize database 101. A prize ID is an ID that identifies each individual prize, or an ID that identifies the type of prize. For example, a prize ID is a JAN (Japan Article Number code) code. Specifically, the promotional image acquisition unit 300 acquires catalog data and extracts the promotional images contained therein. In this embodiment, "promotional image" can be any image prepared in advance to show a prize. It may be a photograph or an illustration that closely resembles the actual item. The promotional images are stored in the promotional image storage unit 332 included in the training data storage unit 330, in a state associated with the prize ID. Note that "prize" refers to "an item that can be won in the crane game device 104."

[0035] The prize database 101 (see Figure 1) is a database that stores information on items that can be won in the crane game machine 104. The operator of the prize database 101 may be the same as or different from the store support server 102. It may also be operated by the prize seller. The prize database 101 provides stores and other places with information about prizes. The catalog data held by the prize database 101 includes the name of the prize, the characteristics of the prize, and the price, along with promotional images (hereinafter referred to as "promotional images"), all associated with the prize ID. The store support server 102 retrieves the catalog data from the prize database 101 and uses the promotional images contained therein as sample images for training data.

[0036] The model learning unit 302 performs machine learning, for example, using a deep learning neural network. Based on training data in which the prize ID is the output data (target variable) and the advertising image is the input data (explanatory variable), the model learning unit 302 generates a judgment model that identifies the prize ID from the photographed image of the prize.

[0037] The data processing unit 308 generates derived images by processing the promotional images. A "derived image" is an image that has been modified based on the promotional image. The derived images are used as training data. In other words, the training data is "inflated." More details will be described later in Modification Example 1.

[0038] The data acquisition unit 312 acquires various data from the crane game machine 104, the store terminal 109, and the user terminal 110. The data provision unit 314 provides various data to the crane game machine 104, the store terminal 109, and the user terminal 110. For example, the data provision unit 314 provides a judgment model to the crane game machine 104.

[0039] The data storage unit 164 includes a training data storage unit 330, a judgment model storage unit 336, a store data storage unit 338, a prize data storage unit 340, and a player data storage unit 342.

[0040] The training data storage unit 330 stores training data used for machine learning. The advertising image storage unit 332 and the derived image storage unit 334 are part of the training data storage unit 330. The advertising image storage unit 332 stores advertising images associated with prize IDs. The derived image storage unit 334 stores derived images associated with prize IDs. Both are used as training data.

[0041] The judgment model storage unit 336 stores the generated judgment model. The store data storage unit 338 stores data about the store (store ID, store name, address, telephone number, email address, and business hours, etc.). The store data storage unit 338 also stores information about the crane game machine 104 installed in each store (for example, the name of the crane game machine 104 (the name given by the store), game machine ID, model, address information for communication destinations, and prize IDs of the prizes it holds, etc.). The prize data storage unit 340 stores data about the prizes. The prize data storage unit 340 stores, for example, catalog data (prize name, prize characteristics, price, and promotional image, etc.). The player data storage unit 342 stores data about the player (user).

[0042] Figure 4 is a functional block diagram of the crane game machine 104. The CPU and other components of the crane game machine 104 are the same as those in the case of the store support server 102. The crane game machine 104 may have a computer, or it may have a circuit board that performs functions equivalent to a computer.

[0043] The crane game device 104 includes a user interface processing unit 180, a mechanism unit 182, a data processing unit 186, a communication unit 184, and a data storage unit 188. The user interface processing unit 180 accepts operations from the player via various input devices and is responsible for processing related to the user interface, such as image display and sound output. The mechanism unit 182 drives various mechanisms such as the crane 118 and the prize stock space 132. The communication unit 184 is responsible for communication processing with the in-store server 107, the store support server 102, the store terminal 109, and the user terminal 110. The data storage unit 188 stores various data. The data processing unit 186 executes various processes based on the data input from the user interface processing unit 180, the data received by the communication unit 184, and the data stored in the data storage unit 188. The data processing unit 186 also functions as an interface to the mechanism unit 182, the user interface processing unit 180, the communication unit 184, and the data storage unit 188.

[0044] The user interface processing unit 180 includes an input unit 152 that receives input from the player and an output unit 190 that outputs various information such as images and sounds to the player. The input unit 152 corresponds to the operation unit 142 (see Figure 2). The output unit 190 also performs image display on the setting display unit 144, etc.

[0045] The communication unit 184 includes a transmission unit 194 that sends various data to the store support server 102 and the like, and a reception unit 196 that receives various data from the store support server 102 and the like.

[0046] The mechanism unit 182 includes a crane drive unit 148 and a camera 122 (image capture unit). As described above, the camera 122 installed in the crane game device 104 captures images of the prize placement table 116 (game field). These captured images are typically used to understand the placement and movement of prizes. In this embodiment, the captured images are used to identify prizes. As described above, the crane drive unit 148 performs the movement of the crane 118 and the gripping and releasing of the arm 146.

[0047] The data processing unit 186 includes a crane control unit 154, a movement determination unit 156, an item identification unit 404, a data acquisition unit 412, and a data provision unit 414.

[0048] The crane control unit 154 instructs the crane drive unit 148 to move, grip, and release according to the operation instructions from the operation unit 142. The movement determination unit 156 determines whether the prize P has fallen into the drop-off opening 130, in other words, whether the crane game is successful or not.

[0049] The item identification unit 404 identifies the prize ID of the prize stored inside the crane game machine 104 based on the captured image of the prize and the judgment model. The item identification unit 404 includes an image extraction unit 405 and a non-target object determination unit 406.

[0050] The image extraction unit 405 extracts a portion of the captured image from the entire image. Hereafter, the entire captured image will be referred to as the "whole image," and a portion of the captured image will be referred to as the "partial image." Both the whole image and the partial image are still captured images. The extraction process for the partial image will be described later in relation to Figures 7 and 8.

[0051] The non-target object determination unit 406 determines whether the subject included in the captured image is a non-target object such as a decoration other than a prize. Non-target objects will be described later in Modification 2. The data acquisition unit 412 acquires various data from the store support server 102 and the like. For example, the data acquisition unit 412 acquires a determination model from the store support server 102. The data provision unit 414 provides various data to the store support server 102 and the like.

[0052] The data storage unit 188 includes a judgment model storage unit 436. The judgment model storage unit 436 stores judgment models obtained from the store support server 102.

[0053] The data storage unit 188 also stores information related to the crane game's program, as well as settings and game play results.

[0054] Figure 5 shows an example of 500 promotional images. In this embodiment, instead of taking photographs of the prizes to create training data, promotional images included in the prize catalog data are used. All currently sold and newly released prizes are always featured in the catalog data. Therefore, promotional images can be obtained without omission. By using catalog data, the effort of photographing the actual prizes is eliminated.

[0055] An example of a promotional image for prize A is shown. The catalog data includes a front promotional image 500a taken from the front, a left side promotional image 500b taken from the left, a back promotional image 500c taken from the rear, a right side promotional image 500d taken from the right, a top promotional image 500e taken from above, and a bottom promotional image 500f taken from below. These correspond to the front view, left side view, rear view, right side view, top view, and bottom view in orthographic projection, respectively.

[0056] The promotional images capture all sides of prize A in at least one of them. Furthermore, since the purpose is sales promotion, the image quality is good. Therefore, they are suitable as sample images for training data.

[0057] Figure 6 is a flowchart showing the processing steps of the store support server 102 during the learning phase. When a new prize A is released, the store support server 102 retrieves an advertisement image of prize A associated with its prize ID from the prize database 101. The store support server 102 then adds the retrieved advertisement image to training data and performs machine learning to generate a judgment model that can recognize prize A. The store support server 102 transmits the judgment model to each crane game machine 104 in each store. The crane game machine 104 updates itself with the received judgment model and becomes capable of recognizing prize A.

[0058] As part of the store support server 102's processing, the advertising image acquisition unit 300 acquires advertising images associated with prize IDs from the prize database 101 (S10) and stores them in the advertising image storage unit 332. The model learning unit 302 uses the advertising images associated with prize IDs in the advertising image storage unit 332 as training data to perform machine learning and generate a judgment model (S12). The generated judgment model is stored in the judgment model storage unit 336. The data provision unit 414 provides the judgment model to each crane game machine 104 (S14).

[0059] Let's move on to the explanation of the operation phase. In the operation phase, let's assume that store X purchases prize A and puts it into crane game machine 104 (S machine). Crane game machine 104 (S machine) takes pictures of its interior. The captured images include prize A. When this image is input into the judgment model of crane game machine 104 (S machine), the prize ID of prize A is output. This allows the system to recognize prize A and determine that "prize A has been put into S machine." Crane game machine 104 (S machine) sends the newly identified prize ID to the store support server 102, which then understands that "prize A is in store X's S machine" and manages this information. This information is referenced by the store terminal 109 and the user terminal 110. In addition to confirming that "prize A is in store X's S machine," the store support server 102 also allows users to view promotional images of prize A.

[0060] This allows store operators and players (users) to obtain a concrete image of prize A, along with its location. Operators can check the prizes stored in machine S at any time by searching on store terminal 109 without having to look inside machine S, thus avoiding disrupting gameplay by going to machine S. Users can know in advance that they can aim for prize A at store X's machine S. This eliminates the need to visit multiple stores to search for prize A, making it convenient for users who want prize A.

[0061] Figure 7 is a schematic diagram showing the first method for extracting a partial image 602 from the overall image 600. Each crane game machine 104 performs processing to identify the prize using the provided judgment model. Here, it is assumed that prize A is contained in crane game machine 104 (machine S) in store X. This figure shows an image (overall image 600) obtained by a ceiling camera 122 that photographs the play space S from above. In the illustrated overall image 600, prize A is included in the group of prizes stacked on the prize display stand 116.

[0062] The image extraction unit 405 extracts multiple partial images 602 from the overall image 600. Each partial image 602 is input to the judgment model. In the first method, as shown in the figure, multiple partial images 602 are obtained by dividing the overall image 600 with a grid pattern. The grid pattern consists of vertical and horizontal lines arranged at equal intervals. Each square portion enclosed by adjacent vertical and adjacent horizontal lines becomes a partial image 602. The length of both the vertical and horizontal sides of the partial image 602 is L (pixels). This figure shows an example in which a partial image 602a containing prize A is extracted. The item identification unit 404 inputs the partial image 602a to the judgment model and performs a judgment calculation using the judgment model to output the prize ID of prize A. The process of inputting a partial image 602 to the judgment model and performing a judgment calculation is sometimes expressed as "applying the partial image 602 to the judgment model."

[0063] As illustrated, if the entire prize A is included in the partial image 602a, it is easily recognized. On the other hand, images of prizes that lie on boundaries of vertical or horizontal lines are difficult to recognize. If prize A straddles a vertical or horizontal line, it may not be recognized properly. In order to recognize prizes that lie on boundaries, it may be possible to perform multiple divisions using multiple grid patterns to generate more partial images 602.

[0064] For example, two methods of division are possible: a two-part division method and a four-part division method. In the two-part division method, the second grid pattern is shifted horizontally by L / 2 (pixels) and vertically by L / 2 (pixels) compared to the first grid pattern. This makes it easier to recognize prizes located near the intersections of vertical and horizontal lines in the first grid pattern. In the four-part division method, the second grid pattern is shifted horizontally by L / 2 (pixels) compared to the first grid pattern. The third grid pattern is shifted vertically by L / 2 (pixels) compared to the first grid pattern. The fourth grid pattern is shifted horizontally and vertically by L / 2 (pixels) compared to the first grid pattern. In this case, prizes that span vertical lines and prizes that span horizontal lines also become easier to recognize.

[0065] The first method is suitable for a ceiling-mounted camera 122 that can photograph the prize group from above at approximately equal distances. This is because prizes are less likely to be obscured by the shadows of other prizes, and image distortion due to perspective differences is small. In the first method, it is easy to recognize the prizes placed on the prize display stand 116 as a whole.

[0066] Figure 8 is a schematic diagram illustrating a second method for extracting a partial image 602 from the overall image 601. In the second method, the prize grasped by the arm 146 is recognized. In other words, the item identification unit 404 identifies the prize targeted by the player (hereinafter referred to as the "target prize"). This figure shows an overall image 601 obtained by the camera 122 which photographs the play space S from diagonally above.

[0067] The item identification unit 404 obtains the three-dimensional position of the arm 146 from the crane control unit 154 when the arm 146 lifts the prize. Based on the three-dimensional position, the item identification unit 404 identifies the position of the tip of the arm 146 in the overall image 601. Alternatively, the item identification unit 404 identifies the position of the tip of the arm 146 by image recognition. Then, the item identification unit 404 determines the range of the partial image 602 based on the position of the tip. The position of the tip is aligned approximately to the center of the partial image 602. The size of the partial image 602 is such that it includes the entire prize held by the arm 146. The image extraction unit 405 extracts the partial image 602 from this range. In this example, the partial image 602b includes prize A held by the arm 146.

[0068] Although not shown in the diagram, the prize may be recognized when the tip of the arm 146 approaches it at its lowest position. In this case as well, the item identification unit 404 will identify the target prize.

[0069] In this case, when the arm 146 is lowered to its lowest position, the item identification unit 404 obtains the three-dimensional position of the arm 146 from the crane control unit 154. As described above, the item identification unit 404 identifies the position of the tip of the arm 146. Then, the image extraction unit 405 identifies the range of the partial image 602 based on the position of the tip and extracts the partial image 602 from that range. In this way, the partial image 602 will include the prize that the tip of the arm 146 touches.

[0070] When photographed from above, the crane 118 overlaps with the prize, making it difficult to capture the prize being held. On the other hand, when photographed from diagonally above, the target prize is less likely to be hidden by the shadow of the crane 118's body. Therefore, when extracting a partial image 602 using the second method, the rear camera 122, which photographs the area near the tip of the arm 146 from diagonally above, is suitable.

[0071] Figure 9 is a flowchart showing the processing steps of the store support server 102 during the operational phase. When the crane game machine 104 identifies a prize, the data provision unit 414 of the crane game machine 104 provides the detection data (including the store ID, game machine ID, and prize ID) to the store support server 102. The game machine ID identifies the crane game machine 104 that detected the prize. The store ID identifies the store where the crane game machine 104 is installed. The prize ID identifies the detected prize. For example, the message conveyed is, "Prize A is in machine S (crane game machine 104) at store X."

[0072] The data acquisition unit 312 of the store support server 102 acquires detection data (including store ID, game machine ID, and prize ID) (S16). The store ID, game machine ID, and prize ID are associated and stored in the store data storage unit 338. This allows the store support server 102 to manage "which prize is in which crane game machine 104 in which store." The detection data configuration of the store data storage unit 338 will be described later in relation to Figure 10.

[0073] The data provision unit 314 of the store support server 102 provides prize information to the store terminal 109 (S18). Specifically, first, the transmission unit (not shown) of the store terminal 109 sends a prize inquiry including the store ID to the store support server 102. When the receiving unit 168 of the store support server 102 receives the prize inquiry, the data provision unit 314 refers to the store data storage unit 338 to identify the game machine ID of the crane game machine 104 installed in that store. The data provision unit 314 uses the store ID and game machine ID pair as a condition key to search the detection data in the store data storage unit 338 and identify the prize ID corresponding to the store ID and game machine ID pair. For each game machine ID, the data provision unit 314 generates a prize list of the prizes stored therein. The prize list includes the name of the prize, the characteristics of the prize, the price, and promotional images. The data provision unit 314 then provides the screen data of the prize list to the store terminal 109. The receiving unit (not shown) of the store terminal 109 receives the prize list screen data, and the display unit (not shown) of the store terminal 109 displays the prize list screen. The operator can view the prize list screen and confirm the prizes contained in each crane game machine 104 in the store. The operator can see the promotional images, so they can visualize the prizes concretely.

[0074] Furthermore, the data provision unit 314 of the store support server 102 also responds to inquiries from the user terminal 110 and provides information about the prizes (S18). Players (users) may want to know what prizes are available in the crane game machine 104 at a specific store, such as a nearby store. Players (users) can specify a store and inquire about the availability of prizes.

[0075] As a specific procedure, when the receiving unit 168 of the store support server 102 receives a store inquiry from the user terminal 110, the data provision unit 314 provides the user terminal 110 with screen data of a store list for selecting a store. When the data acquisition unit (not shown) of the user terminal 110 acquires the screen data of the store list, the display unit (not shown) of the user terminal 110 displays the store list screen. The reception unit (not shown) of the user terminal 110 accepts the selection of a store on this screen. The transmission unit (not shown) of the user terminal 110 sends a prize inquiry including the store ID to the store support server 102. When the receiving unit 168 of the store support server 102 receives the prize inquiry from the user terminal 110, the data provision unit 314 identifies the game device ID, searches the detection data of the store data storage unit 338, and identifies the prize ID, similar to the case of an inquiry from the store terminal 109. The data provision unit 314 generates a prize list and provides the screen data of the prize list to the user terminal 110. However, information not disclosed to the user, such as price, is excluded. The receiving unit (not shown) of the user terminal 110 receives the prize list screen data, and the display unit of the user terminal 110 displays the prize list screen. The player (user) can look at the prize list screen and confirm the prizes stored in the crane game machine 104 in the store. Similar to the operator, the player (user) can visualize the prizes from the images and understand the prize storage status.

[0076] The user terminal 110 may also be able to specify not only the store (store ID) but also the crane game machine 104 (game machine ID) within the store at the same time. In that case, the data provision unit 314 of the store support server 102 performs a search using the specified store ID and game machine ID pair as a condition key.

[0077] When a player (user) wants a specific prize, they want to know which store they can go to to try and win that prize. They might even go to a store they've never visited before specifically to get that prize. Therefore, it would be good to allow the user terminal 110 to inquire about stores that have the specific prize.

[0078] As a specific procedure, when the receiving unit 168 of the store support server 102 receives a prize inquiry from the user terminal 110, the data provision unit 314 provides the user terminal 110 with screen data of the prize list for selecting a prize. When the receiving unit (not shown) of the user terminal 110 receives the screen data of the prize list, the display unit (not shown) of the user terminal 110 displays the prize list screen. The reception unit (not shown) of the user terminal 110 accepts the selection of a prize on this screen. The transmitting unit (not shown) of the user terminal 110 sends a store inquiry including the prize ID to the store support server 102. When the receiving unit 168 of the store support server 102 receives the store inquiry, the data provision unit 314 searches the detection data of the store data storage unit 338 using the prize ID included in the store inquiry as a condition key to identify the pair of store ID and game device ID corresponding to the prize ID. The data provision unit 314 generates a store list based on the identified store ID. The store list displays data such as the store name, address, phone number, email address, and business hours corresponding to the store ID. It also displays the name (as assigned by the store) and model of the crane game machine 104, which is identified by the game machine ID, for each store. The data provision unit 314 provides this store list screen data to the store terminal 109. When the acquisition unit (not shown) of the user terminal 110 acquires the store list screen data, the display unit (not shown) of the user terminal 110 displays the store list screen. The player (user) can then visit a store that has a crane game machine 104 containing the desired prize and enjoy playing as they wish.

[0079] The system may also allow players to check if a specific store (Store X) has a specific prize (Prize A). In this case, the user terminal 110 sends a query specifying the store (Store ID) and prize (Prize ID). The store support server 102 receives this, and the data provision unit 314 performs a search using the specified store ID and prize ID pair as a condition key. The store support server 102 then provides the user terminal 110 with information on the crane game machine 104 (S machine) that has that prize (Prize A) in that store (Store X), according to the searched game machine ID. In this way, the player (user) can, for example, find out if a nearby store has the prize they want, and if so, which crane game machine 104 they should go to.

[0080] Figure 10 is a data configuration diagram of the store data storage unit 338. The store data storage unit 338 of the store support server 102 stores data indicating the prizes identified by the judgment model in the crane game machine 104. The store data storage unit 338 stores the store ID, game machine ID, and prize ID in association. Furthermore, the detection date and time may also be associated. The detection date and time may be added to the detection data by the crane game machine 104 or set by the store support server 102. The detection date and time indicates that the prize was stored at least at that time. Prizes won by players after the detection date and time will remain in the data, but they are not necessarily stored indefinitely. For example, if a prize with prize ID=M0141 is identified in a crane game machine 104 with game machine ID=L0074 located in a store with store ID=K005, the data provision unit 414 of the crane game machine 104 (L0074) provides store ID=K005, game machine ID=L0074, and prize ID=M0141 to the store support server 102. The data acquisition unit 312 of the store support server 102 records store ID=K005, game machine ID=L0074, and prize ID=M0141 in association with the store data storage unit 338. If the prizes in the crane game machine 104 (L0074) are changed and prize ID=M0152 is placed in place of prize (M0141), the camera 122 photographs prize (M0152). The item identification unit 404 identifies the prize (M0152) using a determination model, and the data provision unit 414 provides the store ID=K005, game device ID=L0074, and prize ID=M0152 to the store support server 102. The data acquisition unit 312 of the store support server 102 updates the store ID=K005, game device ID=L0074, and prize ID=M0141 to change them to store ID=K005, game device ID=L0074, and prize ID=M0152.

[0081] When the store support server 102 provides screen data to the store terminal 109 or user terminal 110 as described above, it may filter the data based on the detection date and time. Only data related to prizes detected within a predetermined time period (for example, 24 hours) from the present may be provided. Alternatively, the detection date and time may be added to the screen data so that the detection date and time of each prize is displayed on the store terminal 109 or user terminal 110. For example, suppose data for a prize detected 48 hours ago is recorded. However, when the store terminal 109 or user terminal 110 accesses this data, it is not guaranteed that the prize is still in possession. If the prize was acquired by a player 46 hours ago, it may have already been taken away and is no longer available. Thus, since it is not possible to guarantee that the prize is still in possession, it makes sense to use only data that is highly likely to be in possession.

[0082] This embodiment is summarized below. The store support server 102 creates a judgment model using catalog data. The crane game machine 104 applies captured images of the inside of the machine to the judgment model to identify the prizes contained within. Since catalog data is used, there is no need to take photographs to create training data.

[0083] Information about the prizes contained in the crane game machine 104 is registered in the store support server 102. Store operators and users (players) can find out what prizes are in the crane game machine 104 even when they are away from it. They can also find out which crane game machine 104 contains a particular prize.

[0084] [Example 1] This explains why a large number of sample images are needed in the training data for creating a judgment model. The prizes placed inside the crane game machine 104 are in various positions. Therefore, the prizes included in the captured images are seen from various angles. Also, the lighting conditions inside the crane game machine 104 are different each time. As a result, the colors of the prizes included in the captured images are slightly different. In other words, the images of the prizes are diverse. Therefore, in order to stably recognize the prizes in this way, it is necessary to train the model in advance using many sample images of each prize to represent images of the prizes in various states.

[0085] Even if the advertising images described in the embodiment are used as sample images, the accuracy of the judgment model will improve if there are more sample images. In Modification 1, the advertising images are processed to generate derived images. Then, the derived images are added to the training data as sample images. Various methods can be considered for obtaining derived images by modifying the advertising images as a base. For example, methods include adding noise (such as white noise), rotating the prizes, or flipping them horizontally or vertically.

[0086] The promotional images obtained in this embodiment have good image quality, making them suitable for such modifications.

[0087] In Modification 1, several unique processing methods are proposed to further increase the variations in modifications. The data processing unit 308 (Figure 3) of the store support server 102 generates derived images by processing the advertising images using the various processing methods shown below. Furthermore, the data processing unit 308 generates (updates) training data using the generated derived images as input data (explanatory variables) and the same prize ID as the original advertising image as output data (target variable). The model learning unit 302 uses this training data to perform machine learning and generate a judgment model.

[0088] Figure 11 shows the first example of a derived image. The promotional image 500a is a frontal image of prize A, taken from the front, as explained in relation to Figure 5. The background of the promotional image 500 is often a single color. The background may change for each prize from the perspective of the prize's appearance and image. The data processing unit 308 crops the promotional image 500a to retain only the image of the prize area 501a. The prize area 501a refers to the area inside the outline of the prize. The data processing unit 308 prepares multiple background images 502 in advance. The background image 502 may be an image that combines common colors (for example, beige, which is often used for stuffed animals) as the color of the prize. Here, an example of a camouflage pattern image is shown. The background image 502 may also be an image of the game field taken by the camera 122 installed on the crane game device 104. In this case, the operator inputs instructions for a shooting operation via the setting display unit 144 when loading the prize P into the prize storage unit 114, and the game field is photographed. However, the control unit may also set it to take a photograph when predetermined conditions (such as at regular intervals, when a prize is acquired, or when the door is opened or closed) are met. The captured background image 502 is transmitted to the store support server 102 via the communication unit 184. By doing so, the accuracy of identifying the prize P can be improved by using a background image that shows the game field on which the prize P is actually placed.

[0089] The data processing unit 308 uses the prize area 501a (image) as the upper layer and the background image 502 as the lower layer to perform layer synthesis and generate a derived image 504. The data processing unit 308 performs layer synthesis on multiple background images 502 to generate multiple derived images 504.

[0090] It is desirable to use the same background image 502 for each of the 500 promotional images. This will prevent the model from learning that the image features of background image 502 are a requirement for identifying the prize (the misunderstanding that the camouflage pattern in Figure 11 belongs to prize A). If the same background (camouflage pattern) appears for different prizes (prizes A, B, ...) during the learning process, the judgment model will learn to ignore the image features of the background, as they are not relevant to determining the prize. In other words, the background (camouflage pattern) features that appear for all prizes are treated as not being a "clue" for distinguishing between prizes (prizes A, B, ...). As a result, the judgment model learns to correctly identify the prize by focusing on the image features of the prize itself (prize A) rather than the background.

[0091] Thus, the derived image of the first example is obtained by the data processing unit 308 processing the background image 502 other than the prize area 501a in the advertising image 500. Here, "background image other than the prize area" refers to the image outside the outline of the prize. The background image 502 is adjacent to the outline of the prize. Therefore, the user perceives the background image 502 as representing an object behind the prize.

[0092] Figure 12 shows a second example of a derived image. Prizes placed haphazardly on the prize display stand 116 often overlap each other. For example, a prize placed on top may obscure part of a prize placed below, resulting in a portion of the lower prize being missing from the captured image. It is desirable that the judgment model be able to identify the lower prize even in such captured images. In the second example, a situation is simulated in which part of a prize is missing from the image because it is being covered by something. The object placed on top is not limited to other prizes. Other objects may also get in the way. For example, decorations or the crane 118 may get between the camera 122 and the prize, partially obstructing the image.

[0093] The promotional image 500a is the same as in the first example. The data processing unit 308 prepares multiple foreground images 506 in advance. The foreground image 506 may be, for example, a simple monochrome shape. In this example, a black rectangle is used as the foreground image 506. The foreground image 506 may also be a partial image of another prize. Alternatively, the foreground image 506 may be a partial image of an item such as a decoration or a crane 118.

[0094] The data processing unit 308 generates a derived image 508 by performing layer synthesis, using the advertising image 500a as a lower layer and the foreground image 506 as a higher layer. The placement of the foreground image 506 is arbitrary, but it is best not to overlap too much with the prize area 501a of the advertising image 500a, as this would erase the visual features of prize a. The data processing unit 308 generates multiple derived images 504 by performing layer synthesis on multiple foreground images 506. The placement of the foreground images 506 may be changed each time.

[0095] In this way, the classification model learns to identify the prize using only the image features of the complete portion of the prize included in the promotional image 500a (the remaining part of prize A excluding its left foot). This ensures that even if part of the prize is missing in the actual captured image, the classification model can correctly identify that prize.

[0096] Similar to the background image 502 in the first example, it is desirable to use the same foreground image 506 for each advertising image 500 in the second example as well.

[0097] Thus, the derived image of the second example is obtained by the data processing unit 308 overlaying the foreground image 506 onto a portion of the prize area 501a in the advertising image. Here, "a portion of the prize area" means a part (but not the whole) of the inside of the prize's outline. The "foreground image" is an image that obscures a portion of the prize. Therefore, the user perceives the foreground image 506 as representing an object in front of the prize.

[0098] In the case of prizes that are finely crafted or have long hair, they may be placed in the crane game machine 104 while still wrapped in a plastic bag or similar material. This is to prevent damage from snagging on the details or hair. Plastic bags reflect light. As a result, the reflection of the plastic bag may appear as a white dot in some parts of the prize in the captured image, and parts of the prize's surface may be obscured by the gloss of the plastic bag. It is also possible to simulate this condition. In that case, an image that mimics the reflection of the packaging material (hereinafter referred to as the "reflection image") is used as the foreground image. A reflection image showing the gloss of the prize with a protective sheet attached may also be used as the foreground image. The glossy parts included in the reflection image should be images with a brightness or lightness that is at least higher than the surface of the prize.

[0099] Figure 13 shows a third example of a derived image. The crane game machine 104 often contains multiple types of prizes simultaneously. For example, suppose the crane game machine 104 contains multiple types of prizes, including prizes A and B (stuffed animals). If prize B slips under prize A, part of prize B (for example, its face) may be visible from the side of prize A. Also, if prize B is placed on top of prize A, part of prize B may hide part of prize A. It is also possible to simulate such situations.

[0100] The advertising image 500a is the same as in the first example. The advertising image 518 is an image of prize B, which is not the subject of training. The data processing unit 308 crops the advertising image 500a to leave only the prize area 501a of prize A. The data processing unit 308 also crops the advertising image 518 to leave only the prize area 501b of prize B.

[0101] The data processing unit 308 uses the prize area 501a (image) of prize A as the upper layer and the prize area 501b (image) of prize B, which has been moved slightly, as the lower layer to perform layer synthesis and generate a derived image 516. The prize area 501a (image) of prize A may also be moved. In this way, an image is obtained in which part of prize B is peeking out from the side of prize A. Prize B can also be said to be a type of background image of prize A.

[0102] Although not shown in the diagram, prize B may be positioned as the foreground image of prize A. The data processing unit 308 uses the prize region 501b (image) of prize B as the upper layer and the prize region a (image) of prize A as the lower layer to perform layer synthesis and generate the derived image 516. In this way, an image is obtained in which prize B overlaps with prize A, and a part of prize B hides a part of prize A.

[0103] The data processing unit 308 may generate a derived image 516 in which prize A and prize B do not overlap. However, only a portion of prize B, which is not the subject of training, is included. For example, prize B may be partially cut off.

[0104] Regarding the selection of prize B to be combined with prize A, the data processing unit 308 may pre-determine the prizes to be combined. Prizes from the same series or prizes from the same category (for example, stuffed animals) may be used as the combination targets. Alternatively, the unit may refer to the store data storage unit 338 to select prizes that are actually simultaneously housed in the crane game machine 104. When generating a dedicated judgment model for a particular crane game machine 104 (S machine), the prizes housed in that crane game machine 104 (S machine) may be used as the combination targets. Instead of prize B, an image of a decorative item (for example, a vinyl ball) may be used as the combination target. The image of the prize or decorative item (example of a second item) to be combined with the prize to be judged (example of a first item) may be one or multiple images. In other words, the promotional image of the prize may include images of multiple items.

[0105] Thus, when the crane game machine 104 contains both the first and second prizes, the data processing unit 308 generates a derived image of the first prize by including an image of the second prize in the promotional image of the first prize. Then, the model learning unit 302 trains a judgment model based on the derived image of the first prize that includes an image of the second prize.

[0106] The judgment model is trained using derived images that approximate the captured image containing other prizes or items. By applying the results of this training with derived images, the model becomes able to correctly recognize prizes in captured images that actually contain such items.

[0107] The processing methods described above may be combined. For example, the data processing unit 308 uses the foreground image 506 shown in the second example as the upper layer, the prize area 501a (image) shown in the first example as the middle layer, and the background image 502 shown in the first example as the lower layer to perform layer synthesis and generate the derived image 504. Alternatively, in the layer synthesis of the third example, the background image 502 may be set as the lowest layer and layer synthesis may be performed.

[0108] [Differentiation 2] The prize storage section 114 of the crane game machine 104 contains various decorative items. For example, Christmas trees, balls placed under the prizes, artificial turf, or cushion balls used to hold prizes that have fallen from the arm 146 may be included as decorations. Alternatively, the contents of a prize in a cardboard box (for example, a figurine) may be placed on the back or elsewhere, not as an object to be won, but for decorative purposes. The object to be won is the cardboard box containing the figurine. In other words, the object to be detected is the cardboard box, and the figurine, which is for decorative purposes, should be ignored.

[0109] If decorative items are included in the photographed image, there is a possibility that the decorative items may be mistakenly recognized as a type of prize. Modification 2 prevents the decorative items from being mistakenly recognized as a prize. Hereafter, items other than the prizes that are to be won, such as decorative items, will be referred to as "non-prizes." In the case of the prize in the cardboard box mentioned above, the contents that are displayed (figures) are non-prizes. Note that the appearance of the cardboard box that is to be won and the contents used for decoration are different, so they can be distinguished by their images.

[0110] Figure 14 shows an example of a partial image that includes an object. Here, we show an example of a cushion ball 604. The cushion ball 604 is fixed to the prize display stand 116. Prize C is assumed to be a cardboard box. The overall image 600 includes the cushion ball 604. Assume that a partial image 602c including the cushion ball 604 is extracted using the first method described in relation to Figure 7. In this example, the partial image 602 is enlarged to match the size of prize C. Prize C is not included in the partial image 602c. Therefore, when the item identification unit 404 inputs the partial image 602c to the judgment model, it is correct that no prize ID is output and that there is no prize. If the judgment model were to output a prize ID at this time, assuming that the subject, the cushion ball 604, corresponds to one of the prizes, then that output would be incorrect.

[0111] In Modification 2, if the object included in the partial image 602 is a non-target object, the item identification unit 404 invalidates the identified prize ID even if the determination model identifies the prize ID of the object. If a non-target object happens to have the image characteristics of a certain prize, the prize ID may be identified from the non-target object. In such cases, even if the prize ID is identified, it must be invalidated.

[0112] There are two ways to invalidate a prize ID. (1) After processing with the judgment model, if the subject is not a target object, the identified prize ID is discarded. (2) If the subject is not an object, the judgment model will not be processed. (1) is a determination method that is performed after detecting an item. (2) is a determination method that does not depend on whether an item has been detected or not.

[0113] In either method (1) or (2), it is necessary to determine that the subject contained in partial image 602c is a non-object. A determination model for detecting non-objects (hereinafter referred to as the "non-object determination model" to distinguish it from the prize determination model described above) is used.

[0114] The non-object image storage unit (not shown) of the training data storage unit 330 stores images of non-objects associated with non-object IDs. The non-object ID identifies non-objects such as Christmas trees, underlay balls, artificial turf, cushion balls 604, and decorative figurines. The images of non-objects are images of Christmas trees and the like, and multiple images are prepared for each non-object. These images are used as sample images for the training data.

[0115] The model learning unit 302 uses images of non-objects associated with non-object IDs in the non-object image storage unit as training data to perform machine learning and generate a non-object determination model. The generated non-object determination model is stored in the determination model storage unit 336. The non-object determination model is then provided to the crane game machine 104.

[0116] The non-object determination unit 406 of the crane game machine 104 receives the partial image 602 as input and obtains a non-object ID output from the non-object determination model. When a non-object ID is output, it is recognized that the partial image 602 contains a non-object. If no non-object ID is output, it is determined that the partial image 602 does not contain any non-objects.

[0117] In the case of method (1) described above, the item identification unit 404 inputs the partial image 602 into the prize determination model and performs a determination calculation using the prize determination model. Suppose it is misidentified and the prize ID of prize X is output. This means that the non-target object P (cushion ball 904) was misidentified and determined to be prize X (teddy bear). Furthermore, the non-target object determination unit 406 included in the item identification unit 404 inputs the partial image 602 into the non-target object determination model and performs a determination calculation using the non-target object determination model. As a result, the non-target object ID of non-target object P (cushion ball 904) is output. In other words, the partial image 602 is judged by both the prize determination model and the non-target object determination model, and each outputs a recognition result. If the prize determination model recognizes it as prize X and the non-target object determination model recognizes it as non-target object P, it is denied that it is prize X. The non-object detection model has high accuracy because it has a small number of candidate non-object types. The probability that the non-object detection model recognized it as a cushion ball (for example, 90%) is higher than the probability of recognition by the prize detection model (for example, 70%), so it can be determined that the recognition of cushion ball 604 was correct and the recognition of prize X was incorrect. In this case, the item identification unit 404 discards the prize ID identified by the prize detection model. The prize detection model and the non-object detection model may each output a probability, and the determination result with the higher probability may be prioritized.

[0118] In the case of method (2) described above, the non-target object determination unit 406 first applies the partial image 602 to the non-target object determination model. As a result, the non-target object ID of the cushion ball 604 is output. If a non-target object ID is output, the prize determination model processing for that partial image 602 is not performed. In other words, the partial image 602 containing the non-target object is excluded from prize determination.

[0119] In this way, if the subject of the captured image is determined to be a non-prize item, and the subject is clearly not a prize, the prize will not be identified. This prevents misidentification of items such as Christmas trees, placemats, artificial turf, cushion balls 604, or decorative figurines as prizes.

[0120] Furthermore, the means for recognizing non-objects does not necessarily have to be a machine learning-based judgment model. A program that determines whether or not an object is non-object based on conditions such as color and shape may also be used. For example, the non-object determination unit 406 may, through program processing, perform edge detection in the partial image 602, identify the region of an item based on the detected edges, and determine that it is a cushion ball if the region is a large circle and its color is similar to that of a cushion ball.

[0121] The time-series information of the image can be used to identify and exclude non-objects based on whether or not there is movement in the object. Another possible method is to detect background differences and exclude areas where there is no movement. For example, the crane game machine 104 can compare past captured images (taken with the same camera and angle of view) stored with the image to be detected (whole image 600 or partial image 602) to recognize stationary non-objects. The non-object determination unit 406 compares the pixel value of the subject in the image to be detected with the pixel value of the same position in the past captured image. If the pixel values ​​are the same, it can be determined that the same object is being photographed. Therefore, if the pixel values ​​match, it can be determined that the subject is a stationary non-object. However, the pixel values ​​may match because the prize remains in the same position, which may lead to the misidentification of a non-moving prize as a non-object. To prevent this misidentification, it is assumed that the prizes in the past captured images do not remain in the same position. For example, it is desirable that past images are those taken before prizes are replaced or rearranged, while the images to be detected are those taken after prizes have been replaced or rearranged. Furthermore, comparing the images to be detected with multiple past images accumulated over a long period of time will improve the accuracy of non-target object detection. In this case, if the non-target object is determined based on the condition that the pixel values ​​match in all comparisons with past images, it is easier to prevent misrecognition.

[0122] [Difference 3] During the operational phase, it is also possible to enhance the learning of the judgment model. When a prize is lifted by the arm 146 and falls during transport, the posture of the prize changes each time. Therefore, the ceiling camera 122 and the rear camera 122 can photograph the prize from various angles. In the third modification, the number of samples is increased by adding images taken by the crane game device 104 to the training data.

[0123] Figure 15 shows an example of a partial image 602 of a target prize. Overall image 600e was taken from above by camera 122 on the ceiling before the game started. Overall image 600e includes prize A, which will be targeted by the player later. However, at this stage, it is not known which prize will be targeted.

[0124] The overall image 601e was also taken from diagonally above by the rear camera 122 before gameplay. At this time, arm 146 is not holding anything. After this, gameplay begins, and prize A is lifted by arm 146.

[0125] Overall image 600f was taken from above during transport. Prize A is missing. Also, crane 118 is moving.

[0126] The overall image 601f was also taken from a diagonal angle above during transport. It shows the arm 146 grasping prize A. The image extraction unit 405 can extract a partial image 602f of prize A using the method described in relation to Figure 8. After this, prize A detaches from the arm 146 and falls.

[0127] The overall image 600g was taken from above after the game was played. Prize A is in a different location than it was originally. Also, prize A is in a different position than it was originally. By comparing this overall image 600g with the overall image 600f taken during transport, the fallen prize A can be identified. For prizes other than prize A, the pixel values ​​do not change between overall image 600g and overall image 600f because they are not moving. On the other hand, the pixel value of the pixel representing prize A changes because it is replaced by the image of prize A. Therefore, the areas where the pixel value changes when comparing overall image 600g and overall image 600f represent prize A. By setting the frame of partial image 602 to include the areas where the pixel value changed in overall image 600g, and cropping the image within that frame, partial image 602g containing prize A can be obtained. Note that the part corresponding to the crane 118 in overall image 600g or 600f is ignored. The pixel values ​​of crane 118 also change, but it is clear that they are not prizes. The part of crane 118 can be identified based on the position information obtained from the crane control unit 154.

[0128] The overall image (601g) was taken from a diagonal angle above after gameplay. Arm 146 is not gripping anything.

[0129] Furthermore, prize A, which is included in the overall image 600e before play, can also be identified using the same method. By comparing the overall image 600e before play with the overall image 600f during transport, prize A before it was lifted can be identified. In other words, the areas where the pixel values ​​change when comparing overall image 600e and overall image 600f represent the target prize A. By setting the frame of partial image 602 to include the areas where the pixel values ​​have changed in overall image 600e, and then cropping the image within that frame, partial image 602e containing prize A can be obtained.

[0130] In this way, it is possible to obtain three partial images 602e, 600f, and 600g from the overall images 600e, 600f, and 600g, and from the overall image 601f. These are sent to the crane game machine 104 to be used as training data. It is assumed that the prize ID is identified in at least one of the partial images 602e, 600f, or 600g. If prize detection fails in any of them, the prize ID is unknown and therefore the image cannot be used as a sample image. Also, if the player wins prize A, partial image 602g cannot be extracted from the overall image 600g after the game. Therefore, prize detection after the game is not performed.

[0131] Figure 16 shows the sequence in modified example 3. The ceiling camera 122 of the crane game machine 104 captures an overall image 600e before play (S20). When the crane control unit 154 causes the arm 146 to perform a gripping operation and moves the crane 118, the ceiling camera 122 and the rear camera 122 capture overall images 600f and 601f during transport, respectively (S22). When the operation of the crane 118 is completed, the ceiling camera 122 captures an overall image 600g after play (S24).

[0132] The image extraction unit 405 of the crane game device 104 compares the overall image 600e (above) before play and the overall image 600f (above) during transport using the method described above, and extracts the partial image 602e before play (S26). The image extraction unit 405 extracts the partial image 602f during transport from the overall image 601f (diagonally above) during transport using the method described in relation to Figure 8 (S28). Furthermore, the image extraction unit 405 compares the overall image 600g (above) after play and the overall image 600f (above) during transport, and extracts the partial image 602g after play (S30).

[0133] The item identification unit 404 applies the pre-play partial images 602e to 602g to the judgment model and performs prize detection three times. If prize detection is successful in at least one of the partial images 602e to 602g, and the prize ID is identified (S32). If prize detection fails in all cases, the process ends at this stage.

[0134] The data provision unit 414 of the crane game machine 104 provides the store support server 102 with three partial images 602e to 602g, each corresponding to a prize ID (S34).

[0135] The data acquisition unit 312 of the store support server 102 acquires three partial images 602e to g associated with the prize ID (S40). The data acquisition unit 312 stores the partial images 602e to g in association with the prize ID in the partial image storage unit (not shown) of the training data storage unit 330.

[0136] The model learning unit 302 uses the partial images 602e~g, which are associated with prize IDs in the partial image storage unit, as training data to perform additional machine learning and generate (update) a judgment model. Alternatively, the model learning unit 302 generates a new judgment model by referring to the advertising image storage unit 332, the derived image storage unit 334, and the partial image storage unit (S42). The data provision unit 414 provides the judgment model to each crane game machine 104 (S44).

[0137] In summary, the image extraction unit 405 extracts images of the prizes contained within (partial images) from the play space image (overall image) captured while playing with the crane game machine 104. The model learning unit 302 uses the prize ID identified by the item identification unit 404 as output data (target variable) and the extracted partial images as input data (explanatory variables) to train a judgment model based on this training data.

[0138] The actual posture of prize A when it is placed in the prize storage unit 114 or when it is grasped by the arm 146 is difficult to predict during the learning phase. Therefore, when preparing training data, a method is used in which various postures are captured. However, the postures that prize A can actually take when it is placed or grasped are somewhat limited. This is because the physically stable states are limited. Considering this, partial images that capture the actual posture are more suitable as sample images in terms of learning efficiency. This is because fewer images are needed and the degree of similarity to the image at the time of recognition is high. Scenes similar to the scene illustrated in Figure 15 will be repeated somewhere in the future. Therefore, if the model is trained with three partial images 602e,f,g, the judgment model will grow to be able to detect postures that were not detected this time. In this way, the burden of initial training can be reduced, and the learning accuracy can be gradually improved while in operation.

[0139] The stage of capturing play space images with the crane game machine 104 is not limited to after the crane game machine 104 has started operation. Images of prizes may be extracted from play space images captured before the start of operation. For example, play space images captured during operational testing in the development stage or shipping inspection in the manufacturing stage of the crane game machine 104 may be used. Alternatively, play space images captured during operational checks or demonstrations conducted after the crane game machine 104 has been delivered to the store may be used. Or, play space images may be captured separately from actual operation or operational checks, primarily for the purpose of generating training data.

[0140] [Differentiation Example 4] For prizes that are always individually packaged, it may be acceptable to determine that a prize is "not a prize" if it is not individually packaged.

[0141] [Difference 5] The range in which arm 146 can grasp is mechanically determined. When play begins, crane 118 moves horizontally (X direction) and vertically (Y direction) above the play space S according to the player's operation. The horizontal range in which it can move is limited. Similarly, the vertical range in which it can move is limited. In other words, crane 118 can move within the horizontal plane range determined by its horizontal range and depth range.

[0142] The item identification unit 404 may extract a partial image 602 targeting the horizontal plane range that the arm 146 can grasp below the play space S. Excluding items that are in a range that the arm 146 cannot grasp (for example, decorative items) from the detection target reduces the risk of misidentifying non-prizes as prizes.

[0143] [Modification 6] Unlike Modification 5, the arm 146 may also detect prizes outside its gripping range. For example, flyers may be attached outward with adhesive tape to the inside of the glass plate on the side of the prize storage section 114. If a prize is accidentally thrown, the string or tag attached to the prize may get caught in the gap between the flyer and the glass plate, get caught in the adhesive tape, and not fall down. In this way, the system can detect prizes that have accidentally fallen outside the gripping range. It can also detect prizes that are not far off course but are in a difficult-to-reach place.

[0144] Anticipating such a situation, the item identification unit 404 may extract a partial image 602 even from areas that cannot be grasped. In this way, for example, it can determine that "prize A is caught on the glass plate on the left," or that "prize A is in a difficult-to-reach location."

[0145] In such cases, the image extraction unit 405 receives a partial image 602 extracted from outside the graspable range and performs calculations using a judgment model, resulting in the output of a prize ID. The data provision unit 414 then provides warning data to the store terminal 109, including identification information of the crane game machine 104 (for example, a name such as "machine 1"), a location identified based on the position of the partial image 602 (for example, "the left glass plate"), identification information of the detected prize (for example, the name of prize A), and an image of the prize. When the data acquisition unit (not shown) of the store terminal 109 acquires the warning data, the display unit (not shown) of the store terminal 109 displays the contents of the warning data. This allows the operator to know that "prize A is stuck on the left glass plate of machine 1" or "prize A is in a difficult-to-reach location." An image of prize A can also be viewed. The operator goes to machine 1, removes prize A which is stuck on the left glass plate, and returns it to the prize display stand 116. Alternatively, prize A could be moved to an easily accessible location. By responding quickly to unforeseen circumstances or situations unfavorable to the player in this way, players can play with peace of mind without feeling anxious or uncomfortable.

[0146] [Difference 7] The target prizes include not only the prizes lifted by the arm 146, but also the prizes that are closest to the tip of the arm 146 at their lowest point. Furthermore, as explained in relation to Figure 15, the target prizes are identified both when they are identified in the partial image 602e which includes the prize before it is lifted, and when they are identified in the partial image 602g which includes the dropped prize.

[0147] Knowing the target prize provides clues for recommending other prizes to that player. For example, it can be inferred that a player prefers prizes from the same series or category. Modification 7 shows a system that identifies the target prize for each player and uses that as a clue to recommend other prizes.

[0148] Figure 17 shows the sequence in modified example 7. The player identification unit (not shown) of the crane game machine 104 identifies the player (S50). For example, the player ID is read from the player card with the player ID registered using the card reader (not shown) provided in the crane game machine 104.

[0149] The item identification unit 404 identifies the target prize by the method described in relation to Figure 8 or Figure 15 (S52). The data provision unit 414 provides target data (including player ID, prize ID of the target prize, store ID, and game device ID) to the store support server 102 (S54).

[0150] The data acquisition unit 312 of the store support server 102 acquires target data (S60). The target data is stored in the player data storage unit 342 (S62). The prize recommendation unit (not shown) of the store support server 102 selects a prize to recommend to the player (hereinafter referred to as "recommended prize") based on the target data (S64).

[0151] Let's say Player T's target prize is a figurine of a villain from movie series U. In that case, we can infer that Player T likes movie series U. Therefore, we should select a different prize from the same series. For example, a figurine of a hero character from the same movie series U would be recommended.

[0152] The prize recommendation unit may refer to prizes that the same player has targeted in the past (past target prizes). For example, the player data storage unit 342 may have stored information that player T previously targeted a figurine of a villain character from another movie series V. Comparing this to the current target data, the commonality is that both are villain characters. Therefore, it can be inferred that player T likes villain characters. They may not have a strong attachment to the movie series itself. In this way, referring to past target prizes improves the accuracy of the inference.

[0153] The prize data storage unit 340 can refer to data (e.g., catalog data) stored in it to determine which series each prize belongs to, which category it belongs to (e.g., villain character), and what other characteristics it has. Furthermore, based on statistically analyzed player demographics (e.g., younger players or anime fans), the system may refer to target prizes for other players belonging to the same demographic.

[0154] The data provision unit 314 searches the detection data of the store data storage unit 338 using the prize ID and store ID pair of the recommended prize as a condition key (S66). If the recommended prize is found in that store, it is found, and the crane game machine 104 (game machine ID) containing the recommended prize is identified. If no match is found, the process returns to selecting a different recommended prize (S64).

[0155] The data provision unit 314 generates a recommended prize guide and provides the screen data of the recommended prize guide to the user terminal 110 (S68). The recommended prize guide includes identification information of the crane game machine 104 (for example, a name such as "machine 1"), identification information of the recommended prize (for example, the name of prize A), and an image of the recommended prize.

[0156] The data acquisition unit (not shown) of the user terminal 110 acquires the recommended prize information (S70), and the display unit (not shown) of the user terminal 110 displays the recommended prize information (S72). The player (user) looks at the recommended prize information screen to find out what prizes have been recommended. They can also find out which crane game machine 104 in the store the prizes are housed in. The player (user) can visualize the recommended prizes from the images.

[0157] In addition to introducing crane game machines 104 within the same store, it may also introduce crane game machines 104 at other stores (for example, nearby stores).

[0158] The system may also inform the player (user) whether the target prize is also available in other crane game machines 104. The player (user) can then try to win the target prize in other crane game machines 104 as well.

[0159] To allow users to understand the placement status of target or recommended prizes, the crane game machine 104 may display overall images 600, 601, or partial images 602 on the user terminal 110. In this case, the data acquisition unit 312 of the store support server 102 acquires overall images 600, 601, or partial images 602 from the crane game machine 104, and the data provision unit 314 provides the images to the user terminal 110.

[0160] [Differentiation 8] When a player wins a prize, the store support server 102 may be notified of the prize won.

[0161] When the movement determination unit 156 of the crane game machine 104 determines that the prize P has fallen into the drop-off opening 130, the data provision unit 414 provides the store support server 102 with acquisition data including the target prize ID, store ID, game machine ID, and player ID for that play.

[0162] The data acquisition unit 312 of the store support server 102 acquires acquired data and stores it in the store data storage unit 338. The store data storage unit 338 may also delete information about prizes that are stored as being held. The information about prizes to be deleted can be identified by the target prize ID, store ID, and game device ID.

[0163] Furthermore, the data acquisition unit 312 of the store support server 102 may store the acquired data in the player data storage unit 342. In this way, it is possible to know which prizes each player has already acquired. For example, it may be possible to avoid recommending prizes that have already been acquired.

[0164] The data analysis department (not shown) may calculate the number of attempts required to win a prize to determine an indicator of ease of winning. By referring to the winning data of many players, it is possible to understand the general ease of winning. Depending on the ease of winning, measures such as changing the packaging can be considered. Alternatively, by referring only to the winning data of a specific player, it is possible to understand that player's skill level. For example, prizes appropriate to each player's skill level can be recommended.

[0165] [Other variations] The method for extracting a partial image 602 from the overall image 600 is not limited to the first method described in relation to Figure 7 and the second method described in relation to Figure 8. The partial image 602 may be extracted by other methods. For example, the image extraction unit 405 may detect the area of ​​the item (prize) from the entire captured image 600 through image recognition processing and use that area (an image inside the outline of the item, excluding the background) as the partial image 600. Alternatively, the image containing that area (including the inside of the outline of the item and the background, for example, a square approximately 1.5 to 3.5 times the size of the area of ​​the item) may be used as the partial image 600. Furthermore, the shape of the partial image 602 is not limited to a square. The shape of the partial image 602 may be, for example, a rectangle or a shape other than a quadrilateral.

[0166] The method of collecting images is not limited to the examples described above. By associating images of prizes (examples of items) collected by any method with prize IDs (examples of item IDs) and using them as sample images for training data, a judgment model can be trained. For example, a judgment model can be trained by assigning prize IDs to images of prizes collected on the internet. Alternatively, a judgment model can be trained by assigning prize IDs to images collected by the user terminal 110 or the store terminal 109.

[0167] The functional blocks of the store support server 102 and the crane game machine 104 may be appropriately transferred. For example, the processing of the store support server 102 in the learning phase may be performed by the crane game machine 104. In that case, the crane game machine 104 will have the same functional blocks as the advertising image acquisition unit 300 and the model learning unit 302. Alternatively, the processing of the crane game machine 104 in the operation phase may be performed by the store support server 102. In that case, the store support server 102 will have the same functional blocks as the item identification unit 404.

[0168] The store support server 102 may be divided into a learning server that primarily performs machine learning and an operation support server that assists in the utilization of data during operation.

[0169] Although this explanation assumes a standard offline crane game machine 104, it can also be applied to online crane games. Furthermore, it can be applied to non-crane-based item acquisition game machines. [Explanation of symbols]

[0170] 100 Game system, 101 Prize database, 102 Store support server, 103 Network, 104 Crane game machine, 106 Internet, 107 In-store server, 109 Store terminal, 110 User terminal, 112 Base, 114 Prize storage area, 116 Prize display stand, 118 Crane, 122 Camera, 124 Door, 126 First area, 128 Second area, 130 Drop-off point, 132 Prize stock space, 134 Prize retrieval exit, 136 Control panel, 138 Coin slot, 140 IC card reader, 142 Control unit, 142a Operation buttons, 142b Operation buttons, 144 Setting display unit, 146 Arm, 148 Crane drive unit, 150 Column, 152 Input unit, 154 Crane control unit, 156 Movement determination unit, 160 Communication unit, 162 Data processing unit, 164 Data storage unit, 166 Transmission unit, 168 Receiving unit, 180 User interface processing unit, 182 Mechanism unit, 184 Communication unit, 186 Data processing unit, 188 Data storage unit, 190 Output unit, 194 Transmission unit, 196 Receiving unit, 300 Promotional image acquisition unit, 302 Model learning unit, 308 Data processing unit, 312 Data acquisition unit, 314 Data provision unit, 330 Training data storage unit, 332 Promotional image storage unit, 334 Derived image storage unit, 336 Judgment model storage unit, 338 Store data storage unit, 340 Prize data storage unit, 342 Player data storage unit, 404 Item identification unit, 405 Image extraction unit, 406 Non-target object determination unit, 412 Data acquisition unit, 414 Data provision unit, 436 Judgment model storage unit, 500 Promotional image, 501 Prize area, 502 Background image, 504 Derived image, 506 Foreground image, 508 Derived image, 600 Overall image, 601 Overall image, 602 Partial image, 604 Cushion ball

Claims

1. An item acquisition game device includes an advertising image acquisition unit that acquires an item ID and an advertising image of the item identified by the item ID from a database that stores information on the items to be acquired, A model learning unit generates a determination model that identifies the item ID from a photograph of the item, based on first training data in which the item ID is the target variable and the advertising image is the explanatory variable. An item identification unit identifies the item ID of an item stored inside the item acquisition game device based on a captured image of the item and the determination model, The system includes a data processing unit that generates derived images by processing the aforementioned advertising image, The model learning unit further trains the judgment model based on second training data in which the item ID corresponding to the advertising image processed as a derived image is the target variable and the derived image is the explanatory variable. In the case where the item acquisition game device contains a first item and one or more second items as items to be acquired, The data processing unit generates a derived image of the first article by including images of one or more second articles in the promotional image of the first article. The information processing system is characterized in that the model learning unit trains the judgment model based on derived images of the first item, including an image of the second item.

2. An item acquisition game device includes an advertising image acquisition unit that acquires an item ID and an advertising image of the item identified by the item ID from a database that stores information on the items to be acquired, A model learning unit generates a determination model that identifies the item ID from a photograph of the item, based on first training data in which the item ID is the target variable and the advertising image is the explanatory variable. An item identification unit identifies the item ID of an item stored inside the item acquisition game device based on a captured image of the item and the determination model, The system includes a data processing unit that generates a derived image by processing the advertising image, overlaying a portion of another item, or decorative item, or crane, which is the target of acquisition and has been photographed by a photographic means, as a foreground image onto a portion of the item area in the advertising image. The information processing system is characterized in that the model learning unit further trains the judgment model based on second training data in which the item ID corresponding to the advertising image processed as a derived image is the target variable and the derived image is the explanatory variable.

3. An item acquisition game device includes an advertising image acquisition unit that acquires an item ID and an advertising image of the item identified by the item ID from a database that stores information on the items to be acquired, A model learning unit generates a determination model that identifies the item ID from a photograph of the item, based on first training data in which the item ID is the target variable and the advertising image is the explanatory variable. An item identification unit identifies the item ID of an item stored inside the item acquisition game device based on a captured image of the item and the determination model, The system includes a data processing unit that generates derived images by processing the aforementioned advertising image, The data processing unit generates the derived image by using a game field image captured by a shooting means provided in the item acquisition game device as a background image, or by using a partial image of another item, decoration, or crane that is the target of acquisition, captured by the shooting means, as a foreground image. The information processing system is characterized in that the model learning unit further trains the judgment model based on second training data in which the item ID corresponding to the advertising image processed as a derived image is the target variable and the derived image is the explanatory variable.

4. An item acquisition game device includes an advertising image acquisition unit that acquires an item ID and an advertising image of the item identified by the item ID from a database that stores information on the items to be acquired, A model learning unit generates a determination model that identifies the item ID from a photograph of the item, based on first training data in which the item ID is the target variable and the advertising image is the explanatory variable. The item acquisition game device includes an item identification unit that identifies the item ID of an item stored inside the item based on a captured image of the item and the determination model, The information processing system is characterized in that, when the item identification unit identifies an item ID of the subject in the captured image, if the subject is a non-target object other than the item to be acquired, the item ID of the subject is invalidated even if the determination model identifies the identified item ID.

5. An item acquisition game device includes an advertising image acquisition unit that acquires an item ID and an advertising image of the item identified by the item ID from a database that stores information on the items to be acquired, A model learning unit generates a determination model that identifies the item ID from a photograph of the item, based on first training data in which the item ID is the target variable and the advertising image is the explanatory variable. An item identification unit identifies the item ID of an item stored inside the item acquisition game device based on a captured image of the item and the determination model, The device comprises an image extraction unit that extracts the captured images of the stored items from the play space image captured by the item acquisition game device, The information processing system is characterized in that the model learning unit further trains the judgment model based on third training data in which the item ID identified by the item identification unit is the target variable and the extracted captured image is the explanatory variable.

Images