Information processing device, information processing method, and recording medium

The information processing device addresses erroneous recognition in interactive content by determining interaction presentation based on recognized results and user actions, ensuring uninterrupted gameplay by managing recognition accuracy and correction intervals.

WO2024241810A1PCT designated stage expired Publication Date: 2024-11-28SONY GROUP CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/015798
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-04-23
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Conventional information processing systems fail to effectively handle erroneous object recognition during interactive content presentation, leading to hindered progress in applications like board games, where misrecognition results in user intervention and task interruption.

Method used

An information processing device that acquires image information of a presentation target, performs recognition processing, and determines whether to present interactions based on the recognized results, user actions, and game rules, using a control unit to manage recognition accuracy and prevent frequent corrections.

Benefits of technology

The system effectively suppresses the impact of misrecognition on content progression by allowing users to correct errors without interrupting the game, ensuring smooth gameplay by determining when to display correction dialogs and adjusting recognition processing intervals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024015798_28112024_PF_FP_ABST
    Figure JP2024015798_28112024_PF_FP_ABST
Patent Text Reader

Abstract

This information processing device is able to present, to a presentation target, content and an interaction pertaining to the progress of the content, and is provided with a control unit. The control unit: acquires image information in which at least the presentation target is in a photographing target range; recognizes the state of an object including a user operation in the recognition target range by executing recognition processing on the recognition target range in the image information; determines, as a result of recognition, whether to present the interaction associated with the recognition result on the basis of the user operation and rule information pertaining to the content; and presents the interaction to the presentation target in accordance with the determined result.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a recording medium.

[0002] Conventionally, there is known a technology for presenting information by projecting arbitrary content onto a tabletop or the like using a projector or the like (see, for example, Patent Document 1). This technology can also recognize, for example, a real object present in the presentation target, and call a function associated with the object according to the recognition result.

[0003] Using this technology, it will be possible to provide game content that allows players to progress through the game by projecting the surface of a board game, for example, onto a table, and outputting interactions such as video and audio based on the recognition results of real-world objects such as dice and pieces.

[0004] JP 2015-090524 A

[0005] However, in the above-mentioned conventional technology, the recognition result of the object recognized by the system is always treated as correct, and the case where an erroneous recognition occurs is not sufficiently considered.

[0006] As a result, if an object is misrecognized, the user has no choice but to ignore the recognition result and continue the task, or to reposition the object and have it recognized again, which can sometimes hinder progress through the content.

[0007] Therefore, the present disclosure proposes an information processing device, an information processing method, and a recording medium that can prevent the progress of content presented to a presentation target from being hindered by erroneous recognition of real objects related to the content.

[0008] In order to solve the above problem, an information processing device according to one embodiment of the present disclosure is an information processing device capable of presenting content and interactions related to progress of the content to a presentation target, and includes a control unit. The control unit acquires image information having at least the presentation target as a capture target range, recognizes a state of an object in the recognition target range including a user's action by performing a recognition process on a recognition target range in the image information, determines whether to present the interaction associated with the recognition result based on the recognition result, the user's action, and rule information related to the content, and presents the interaction to the presentation target according to the determination result.

[0009] 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present disclosure. FIG. 2 is a block diagram illustrating an example of a configuration of an information processing device according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of scoring definition information. FIG. 4 is a diagram illustrating an example of lock definition information. FIG. 5 is a diagram illustrating an example of recognition interpretation information. FIG. 6 is a flowchart illustrating an overall processing procedure executed by the information processing device 10 according to an embodiment of the present disclosure. FIG. 7 is a flowchart illustrating a processing procedure in a recognition processing unit. FIG. 8 is a flowchart illustrating a processing procedure in a determination processing unit (part 1). FIG. 9 is a flowchart illustrating a processing procedure in a determination processing unit. FIG. 10 is an explanatory diagram of a first embodiment. FIG. 11 is a diagram illustrating recognition interpretation information in the first embodiment. FIG. 12 is an explanatory diagram of a second embodiment. FIG. 13 is a diagram illustrating recognition interpretation information in the second embodiment. FIG. 14 is an explanatory diagram of a third embodiment. FIG. 15 is a diagram illustrating recognition interpretation information in the third embodiment. FIG. 16 is an explanatory diagram of a fourth embodiment. FIG. 17 is a diagram illustrating recognition interpretation information in the fourth embodiment. FIG. 18 is an explanatory diagram of a fifth embodiment. FIG. 19 is a diagram illustrating recognition interpretation information in the fifth embodiment. FIG. 20 is an explanatory diagram of a sixth embodiment. FIG. 21 is a diagram illustrating recognition interpretation information in the sixth embodiment. FIG. 22 is an explanatory diagram of a seventh embodiment. FIG. 23 is a diagram illustrating recognition interpretation information in the seventh embodiment. FIG. 24 is an explanatory diagram of an eighth embodiment. FIG. 25 is an explanatory diagram of a ninth embodiment. 10 is a diagram showing recognition interpretation information in the ninth embodiment. FIG. 11 is an explanatory diagram of the tenth embodiment. FIG. 12 is a diagram showing recognition interpretation information in the tenth embodiment. FIG. 13 is an explanatory diagram of the tenth embodiment. FIG. 14 is an explanatory diagram of the fifteenth embodiment. FIG. 15 is an explanatory diagram of the sixteenth embodiment. FIG. 17 is an explanatory diagram of the eighteenth embodiment. FIG. 19 is an explanatory diagram of the twentieth embodiment. FIG. 20 is an explanatory diagram of the twenty-first embodiment. FIG. 22 is an explanatory diagram of the twenty-third embodiment. FIG. 24 is an explanatory diagram of the twenty-fifth embodiment. FIG. 26 is an explanatory diagram of the twenty-seventh embodiment. FIG. 28 is an explanatory diagram of the twenty-ninth embodiment. FIG. 29 is an explanatory diagram of the thirtieth embodiment. FIG. 30 is an explanatory diagram of the thirty-first embodiment. FIG. 32 is an explanatory diagram of the thirty-second embodiment. FIG. 33 is an explanatory diagram of the thirty-third embodiment.Fig. 34 is an explanatory diagram of a thirty-fourth embodiment. Fig. 35 is an explanatory diagram of a thirty-sixth embodiment. Fig. 37 is an explanatory diagram of a thirty-seventh embodiment. Fig. 38 is a hardware configuration diagram showing an example of a computer that realizes the functions of an information processing device.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] In the following description, an information processing device according to an embodiment of the present disclosure (hereinafter referred to as "the present embodiment") is assumed to be the information processing device 10 shown in Fig. 2 and subsequent drawings. An information processing method according to the present embodiment is assumed to be an information processing method executed by the information processing device 10.

[0012] In the following, the content provided by the information processing device 10 is mainly game content. The game content is Monopoly (registered trademark) or Settlers of Catan (hereinafter abbreviated as "Catan"), for example. In the following, "player" is an example of "user."

[0013] The present disclosure will be described in the following order: 1. Configuration example of an information processing system 2. Configuration example of an information processing device 3. Processing procedure 3-1. Overall processing procedure 3-2. Processing procedure in the recognition processing unit 3-3. Processing procedure in the judgment processing unit for Monopoly content 3-4. Processing procedure in the judgment processing unit for Catan content 4. Examples for Monopoly content 4-1. First example 4-2. Second example 4-3. Third example 4-4. Fourth example 4-5. Fifth example 4-6. Sixth example 4-7. Seventh example 5. Examples for Catan content 5-1. Eighth example 5-2. Ninth example 5-3. Tenth example 5-4. Eleventh example 5-5. Twelfth example 6. Examples regarding various corrections in game content 6-1. Thirteenth example 6-2. Fourteenth example 6-3. 15th Example 6-4. 16th Example 6-5. 17th Example 6-6. 18th Example 6-7. 19th Example 6-8. 20th Example 6-9. 21st Example 6-10. 22nd Example 7. Examples Related to Object Recognition 7-1. 23rd Example 7-2. 24th Example 7-3. 25th Example 7-4. 26th Example 7-5. 27th Example 7-6. 28th Example 7-7. 29th Example 7-8. 30th Example 8. Examples Other than Game Content 8-1. 31st Example 8-2. 32nd Example 8-3. 33rd Example 8-4. 34th Example 8-5. 35th Example 8-6. 36th Example 8-7. 37th Example 9. Modifications 10. Hardware Configuration 11. Conclusion

[0014] 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment of the present disclosure. The information processing system according to the present embodiment is configured to present any content, such as game content, to a presentation target of information, and to output interactions such as video and audio according to the recognition result of a real object related to the content.

[0015] As shown in the left diagram of Fig. 1, the information processing system is configured as, for example, a projection type. In the projection type information processing system, the presentation target is a tabletop or the like, and various sensors such as a camera and a projector are provided above the tabletop.

[0016] In the projection type, when the content is game content, the information processing system projects a board image of, for example, Monopoly or Catan onto the table. The information processing system also sets part or all of the table as the recognition target range and performs image recognition processing to recognize real objects, such as dice and pieces, present on the table based on image information of the table captured by a camera. The information processing system then progresses the game while outputting interactions such as visual and audio effects and guidance according to the results of the recognition processing.

[0017] Furthermore, as shown in the right diagram of Fig. 1, the information processing system may be configured as, for example, an eyewear type. In the eyewear type, the information processing system is configured with smart glasses or the like, and presents content superimposed on the real world seen by the user through a transparent display placed in front of the user's eyes.

[0018] The information processing system includes an information processing device 10 (see FIG. 2). An information processing method according to this embodiment using the information processing system is executed by this information processing device 10. Next, an example of the configuration of this information processing device 10 will be described. Note that in this embodiment, the information processing system will be described as being of a projection type.

[0019] 2. Configuration example of information processing device>> Fig. 2 is a block diagram showing a configuration example of an information processing device 10 according to an embodiment of the present disclosure. Note that Fig. 2 shows, in functional blocks, only components necessary for explaining this embodiment, and omits descriptions of general components.

[0020] In addition, in the description using FIG. 2, the description of components that have already been described will be appropriately simplified or omitted.

[0021] 3, the information processing device 10 includes a storage unit 11 and a control unit 12. The information processing device 10 is also connected to a sensor unit 3 and a presentation unit 5. The sensor unit 3 performs various types of sensing to acquire a situation in a space where a certain environment / background exists.

[0022] The situation referred to here includes the situation of the object in the recognition target range, and the situation of the object includes the user's actions, i.e., the operations performed by the user regarding the game while the game is in progress.

[0023] The sensor unit 3 includes a camera, a depth sensor, a touch sensor, etc. The camera is installed so that it captures at least part or all of the tabletop. The camera is typically a visible light camera, an infrared camera, etc.

[0024] Depth sensors sense three-dimensional information. Common depth sensors include stereo cameras, time-of-flight sensors, and structured light sensors.

[0025] Touch sensors sense the movement of the user's fingers, etc. Touch sensors are generally implemented by taking pictures from above or below with a camera in addition to the depth sensor mentioned above, or by placing a touch panel on the table surface.

[0026] The presentation unit 5 presents to the user interactions such as video and audio corresponding to the object recognition result output from the information processing device 10. In this embodiment, the presentation unit 5 is realized by, for example, a projector, a speaker, or the like.

[0027] Alternatively, the presentation unit 5 can be realized by a touch panel, a liquid crystal screen, etc. In the case of the above-mentioned eyewear type, the presentation unit 5 is realized by a display, a speaker, etc. in the smart glasses.

[0028] Although not shown in FIG. 2, the information processing device 10 may be connected to an input device such as a keyboard or a mouse that accepts general input operations from a user or the like.

[0029] The storage unit 11 is realized by a storage device such as a random access memory (RAM), a read only memory (ROM), a flash memory, or a hard disk drive (HDD).

[0030] In the example of FIG. 2, the storage unit 11 stores a recognition model 11a, scoring definition information 11b, locking seconds definition information 11c, a recognition interpretation information DB (DataBase) 11d, and a content DB 11e.

[0031] The recognition model 11a is an image recognition model used in the recognition processing executed by the recognition processing unit 12b, which will be described later. The recognition model 11a is realized as a DNN (Deep Neural Network) model trained using a deep learning algorithm, for example. The recognition model 11a is trained in advance so that it can recognize objects present in a recognition target range included in image information captured by the camera of the sensor unit 3.

[0032] The scoring definition information 11b is information that defines thresholds and the like for determining whether or not to execute a presentation process associated with a recognition result obtained by a recognition processing unit 12b (described later).

[0033] 3 is a diagram showing an example of the scoring definition information 11b. As shown in FIG. 3, the scoring definition information 11b includes a target column, an item column, and a score column. The target column defines various objects to be recognized in game content, for example.

[0034] The item string and score string define, when each object in the target string is recognized, each presentation item, which is a variety of interactions associated with the recognition result, and a threshold value for determining whether or not to present these items.

[0035] In addition, the scoring definition information 11b defines a deduction value that is used by the judgment processing unit 12c (described later) to determine the accuracy of the recognition result and, depending on the score, to determine whether or not to prompt the user to make corrections (see "Deduction value for ignoring correction dialog" in the figure).

[0036] In addition, the scoring definition information 11b defines a penalty value to ensure that when a user places a piece in an area where it is not permitted or places more than the maximum number of pieces, this fact is maintained (see "Placement-prohibited square penalty value" and "Excessive number penalty value" in the figure).

[0037] More specific methods of using the scoring definition information 11b will be described in the explanations of specific examples below.

[0038] Returning to the explanation of FIG. 2 , the locking seconds definition information 11c defines the locking seconds to prevent the recognition processing unit 12b from performing the recognition process again. The locking seconds is defined as a bonus value to maintain the reliability of the recognition processing unit 12b by preventing repeated erroneous recognition when, for example, a user explicitly corrects an interaction based on the recognition result of the recognition processing unit 12b. In other words, the locking seconds is the interval in seconds until the recognition processing is performed again after a recognition process has been performed once. Note that, although the unit is seconds here, it may also be in frames of image information instead of seconds. The locking seconds is stored in the recognition interpretation information, which will be described later.

[0039] 4 is a diagram showing an example of the locking seconds definition information 11c. As shown in FIG. 4, the locking seconds definition information 11c includes a column and a number of seconds.

[0040] As shown in FIG. 4, the locking seconds definition information 11c is defined so that, for example, when a normal recognition process is executed, an interval of 10 seconds is provided before re-recognition is performed on the same area.

[0041] Furthermore, the lock seconds definition information 11c is defined to provide a 10-second interval, since if a correction dialog box is presented to the user and ignored, it is interpreted that the recognition result in the recognition process is incorrect. This interval prevents the user from being presented with correction dialog boxes multiple times and disrupting the progress of the game.

[0042] In addition, the lock seconds definition information 11c is defined to set a relatively large interval of 99 seconds, for example, when the user explicitly makes a correction, so that the user's correction is considered correct regardless of the recognition result in the recognition process.

[0043] In addition, the lock seconds definition information 11c is defined to set an interval of 20 seconds, which is longer than that of normal recognition processing, because, for example, when advanced recognition processing is performed, which takes longer than normal recognition processing but is considered to be more reliable than normal recognition processing.

[0044] A more specific method of using the lock seconds definition information 11c will be described in the explanation of each specific embodiment below.

[0045] Returning to the explanation of Fig. 2, the recognition interpretation information DB 11d is a database of recognition interpretation information that stores values ​​that add or subtract from the recognition results and the user's recognition content in order to supplement any deficiencies in the recognition results without relying solely on the recognition results obtained by the recognition process and to infer the correct situation within the recognition target range.

[0046] 5 is a diagram showing an example of the recognition interpretation information. The recognition interpretation information is generated as information for each recognition area recognized in the recognition process. One line corresponds to one recognition area.

[0047] 5, the recognition interpretation information includes a top left coordinate column, a top right coordinate column, a recognition result column, an interpretation result column, and a lock second column. The top left coordinate column and the top right coordinate column store the top left coordinate and the top right coordinate that define the recognized recognition area.

[0048] The recognition result column stores the recognition result of the recognition process and its accuracy score (hereinafter referred to as "score" as appropriate). If multiple recognition results are obtained for the same area, the multiple recognition results are stored.

[0049] The interpretation result column stores the interpretation result determined based on not only the recognition result in the recognition process but also the user's operation, etc. The locking seconds column stores the number of locking seconds for the recognition process for the corresponding area.

[0050] More specific methods of using the recognition interpretation information will be described in the explanations of specific embodiments below.

[0051] Returning to the explanation of Fig. 2, the content DB 11e is a database that stores content data of various types of content provided by the information processing device 10. For example, if the content is game content, each piece of content data includes rule information that defines the rules of the game.

[0052] The control unit 12 controls each unit of the information processing device 10. The control unit 12 is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphical processing unit (GPU), or the like executing a program according to the present embodiment (not shown) stored in the storage unit 11 using RAM as a work area. The control unit 12 can also be realized by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0053] The control unit 12 has an image acquisition unit 12a, a recognition processing unit 12b, a determination processing unit 12c, and a presentation processing unit 12d, and realizes or executes the functions and actions of information processing described below.

[0054] The image acquisition unit 12a acquires image information, which is sensing data from the sensor unit 3. The recognition processing unit 12b executes recognition processing, which is image recognition, on the image information acquired by the image acquisition unit 12a, using the recognition model 11a.

[0055] The recognition processing unit 12b is configured to be able to execute a first recognition process, which is a normal recognition process, and a second recognition process, which is an advanced recognition process that takes more time than the first recognition process but has higher recognition performance than the first recognition process. The second recognition process is executed based on the determination result of the determination processing unit 12c.

[0056] The determination processing unit 12c determines whether to output an interaction associated with the recognition result based on the accuracy of the recognition result recognized by the recognition processing unit 12b, the recognized user operation, the above-mentioned rule information, etc. At this time, the determination processing unit 12c appropriately determines whether or not it is necessary to perform advanced recognition processing, and if it is determined that it is necessary to perform advanced recognition processing, it causes the recognition processing unit 12b to perform the advanced recognition processing.

[0057] The presentation processing unit 12d outputs an interaction such as a video or sound according to the determination result of the determination processing unit 12c, and causes the presentation unit 5 to present it.

[0058] <<3. Processing Procedure>> Next, processing procedures executed by the information processing device 10 will be described with reference to Fig. 6 to Fig. 9. Fig. 6 is a flowchart showing the overall processing procedures executed by the information processing device 10 according to an embodiment of the present disclosure.

[0059] 6, the control unit 12 of the information processing device 10 determines whether or not an end operation has been performed (step S101). If an end operation has not been performed (step S101, No), the control unit 12 executes the processes from step S102 onward. If an end operation has been performed (step S101, Yes), the control unit 12 ends the process.

[0060] In step S102, the control unit 12 acquires information (image information from the sensor unit 3) from the image acquisition unit 12a. Next, the control unit 12 performs recognition on the acquired information using the recognition processing unit 12b (step S103).

[0061] Furthermore, when the user explicitly corrects the recognition result obtained by the recognition process, the control unit 12 adds the locking number of seconds defined in the locking number of seconds definition information 11c (step S104). This allows the control unit 12 to store information that prevents (locks) the recognition process from being performed again for a certain period of time when the user makes the correction.

[0062] The control unit 12 then determines the accuracy of the recognition result obtained by the recognition processing unit 12b using the determination processing unit 12c (step S105), and compares the score of the determined accuracy with the threshold score for each item defined in the score definition information 11b to determine whether to execute the presentation process (step S106).The control unit 12 also stores the recognition result in the recognition interpretation information DB 11d (step S107).

[0063] The control unit 12 then displays information (i.e., interactions such as effects and guidance according to the determination result in step S106) using the presentation processing unit 12d (step S108). Note that the interactions may include audio. The control unit 12 then repeats the process from step S101.

[0064] <3-2. Processing Procedure in Recognition Processing Unit> Next, Fig. 7 is a flowchart showing the processing procedure in the recognition processing unit 12b. As shown in Fig. 7, the control unit 12 acquires information from the image acquisition unit 12a (step S201). Next, the control unit 12 recognizes the acquired information in the recognition processing unit 12b (step S202). Step S201 corresponds to the above-mentioned step S102, and step S202 corresponds to the above-mentioned step S103.

[0065] Then, the control unit 12 executes a loop process that is repeated for the number of areas detected in the recognition process (step S203). In the loop process, the control unit 12 determines whether the detected area is an area stored in the recognition interpretation information DB 11d (step S204).

[0066] If the area is not saved (step S204, No), it is an area that has been detected for the first time, so the control unit 12 saves new recognition interpretation information corresponding to this area in the recognition interpretation information DB 11d (step S208). On the other hand, if the area is saved (step S204, Yes), it is an area that has been detected before, so the control unit 12 determines whether the pixel values ​​have changed since the previous time (step S205).

[0067] If it has changed (step S205, Yes), the control unit 12 decrements the locking seconds by 1 (step S206) and continues recognition. The control unit 12 also determines whether the locking seconds has become 0 (step S207), and if it has become 0 (step S207, Yes), stores the recognition interpretation information in the recognition interpretation information DB 11d (step S208).

[0068] If the pixel value has not changed from the previous time (No in step S205), the control unit 12 repeats the process from step S204 for the next area. Furthermore, if the locking seconds is not 0 (Yes in step S207), the control unit 12 similarly repeats the process from step S204 for the next area. Furthermore, even after executing step S208, the control unit 12 similarly repeats the process from step S204 for the next area.

[0069] Then, when all detected areas have been processed, the control unit 12 ends the processing in the recognition processing unit 12b.

[0070] <3-3. Processing Procedure in the Determination Processor for Monopoly Content> Next, Fig. 8 is a flowchart (part 1) showing the processing procedure in the determination processor 12c. The determination criteria in the determination processor 12c vary depending on the content. Fig. 8 shows the processing procedure in the determination processor 12c for Monopoly content.

[0071] 8, in the determination process of the determination processing unit 12c for the Monopoly content, the control unit 12 executes a loop process repeated for the number of lines stored in the recognition interpretation information DB 11d (step S301). In this loop process, the control unit 12 determines whether the recognition result has changed since the previous time (step S302). If it has changed (step S302, Yes), the control unit 12 performs processing according to the detected object (dice, house, etc.) (step S303). Then, the control unit 12 stores the processed state according to the object (step S304).

[0072] If the recognition result has not changed since the previous time (No at step S302), the control unit 12 repeats the process from step S302 for the next row. Then, when all rows stored in the recognition interpretation information DB 11d have been processed, the control unit 12 ends the process at the determination processing unit 12c.

[0073] In the case of Monopoly content, in step S303, the control unit 12 determines whether the detected object is, for example, a die or a house. If it is a die (step S303, die), the control unit 12 recognizes the number rolled on the die and displays the destination based on the number rolled on the board (step S401).

[0074] Then, the control unit 12 determines whether the recognized dice roll is correct or not by determining whether the user's operation of advancing the piece and the system's recognition (i.e., the presented destination) have diverged (step S402).

[0075] If a discrepancy occurs (step S402, Yes), the control unit 12 determines whether the accuracy of the recognition result is equal to or greater than 90 points as defined in the scoring definition information 11b (step S403) in order to determine whether to display a dialog informing the user that the location is incorrect.

[0076] If the score is 90 or more (Yes in step S403), the control unit 12 displays a message indicating that the placement is incorrect (step S404).Then, the control unit 12 determines whether the user followed the prompts (the dialogue in step S404) to determine whether the recognition result is truly correct (step S405).

[0077] If the user does not comply with the prompting (No in step S405), the control unit 12 determines that the user did not comply because the recognition result is incorrect. Then, the control unit 12 adds 10 seconds defined in the locking seconds definition information 11c to the locking seconds (step S406) to adjust the time until the recognition process is performed again.

[0078] This prevents the information processing device 10 from frequently making erroneous recognitions and causing confusion for the user. Then, the control unit 12 determines the square placed by the user as correct and updates the interpretation result in the recognition interpretation information (step S407).

[0079] On the other hand, if the user follows the suggestions (step S405, Yes), the control unit 12 determines that the recognition result is correct, and the system-suggested square is regarded as correct, and the interpretation result is not manipulated (step S408). Also, if no discrepancy occurs in step S402 (step S402, No), the control unit 12 determines that the recognition result is correct, and the system-suggested square is regarded as correct, and the interpretation result is not manipulated (step S408).

[0080] If the score is less than 90 points in step S403 (No in step S403), the control unit 12 determines that the square placed by the user is positive, and updates the interpretation result in the recognition interpretation information (step S407).

[0081] Also, if it is determined in step S403 that the object is a house (step S303, if it is a house), the control unit 12 determines whether the area in which the object recognized as a house piece is placed is a square in which a house may be placed (step S501).

[0082] If it is okay to place it (step S501, Yes), to determine whether to present a dialog indicating that a house has been built, it is determined whether the accuracy of the recognition result is 90 points or more as defined in the score definition information 11b (step S502).

[0083] If the score is 90 points or more (step S502, Yes), the control unit 12 displays an effect on the board indicating that the user has built a house (step S503).Then, in order to determine whether the recognition result is truly correct, the control unit 12 determines whether the user has requested a correction to the effect of step S503 (step S504).

[0084] If no request has been made (step S504, No), it is considered that the user has accepted the effect, and the result presented by the system (i.e., the recognition result) is regarded as correct (step S505).

[0085] If so (step S504, Yes), the control unit 12 confirms with the user the correct placement (step S506).The control unit 12 then determines that the user is correcting the recognition result, and sets the locking seconds to 99 seconds defined in the locking seconds definition information 11c so that the control unit 12 can wait until the user's correction process is complete (step S507).

[0086] In this way, when the user explicitly corrects the recognition result of the information processing device 10, there is almost no need for the information processing device 10 to perform the recognition process again, so a relatively large locking time is defined.

[0087] When the user's correction process is complete, the control unit 12 updates the interpretation result by regarding the user's instruction as correct (step S508), and then the control unit 12 displays the effect again to the user (step S509).

[0088] If the score is less than 90 points in step S502 (No in step S502), the processes in steps S506 to S509 are also executed.

[0089] Furthermore, if it is determined in step S501 that the square is not suitable for placing a house (step S501, No), the control unit 12 determines whether to display a dialog indicating that the placement is incorrect, and determines whether the score for the accuracy of the recognition result is still 80 points or more even after deducting 10 points, which is the deduction value for squares that cannot be placed in the scoring definition information 11b (step S601).

[0090] If the score is 80 or more (step S601, Yes), a message is displayed indicating that the user's house layout is incorrect (step S602). In principle, the user's recognition is considered correct, but if the recognition result still seems accurate even after deducting 10 points, the user can be prompted to confirm.

[0091] Then, the control unit 12 determines whether the user has followed the correction instructions (presented in step S602) to confirm whether the determination that the layout of the user's house is incorrect is really correct (step S603).

[0092] If the user complies (step S603, Yes), the control unit 12 deletes the current interpretation result (step S605), and performs recognition processing on the image information of the next frame for the house that has been repositioned in accordance with the user's correction instruction.

[0093] If the user does not comply (step S603, No), the control unit 12 determines that the user has ignored the correction dialog and adds 10 seconds defined in the locking seconds definition information 11c to the locking seconds (step S604). In this case, it is considered that the dialog was ignored because the recognition result was incorrect, so the control unit 12 prevents the recognition process from being performed again immediately.

[0094] If the score is less than 80 points in step S601 (No in step S601), the control unit 12 deletes the current interpretation result (step S605). In other words, step S605 updates the interpretation result so that it is considered that the house has not been built.

[0095] Then, the control unit 12 saves the state after steps S407, S408, S505, S509, and S605 are executed (step S304).

[0096] <3-4. Processing Procedure in the Determination Processing Unit in Catan Content> Next, Fig. 9 is a flowchart (part 1) showing the processing procedure in the determination processing unit 12c in the Catan content. Fig. 9 shows the processing procedure in the determination processing unit 12c in the Catan content.

[0097] Steps S301, S302, and S304 are the same as those in Figure 8. As shown in Figure 9, in the judgment process by the judgment processing unit 12c in the Catan content, the control unit 12 executes a loop process that is repeated for the number of lines stored in the recognition interpretation information DB 11d (step S301). In this loop process, the control unit 12 determines whether the recognition result has changed since the previous time (step S302). If it has changed (step S302, Yes), the control unit 12 performs processing according to the detected object (clear land, city, highway, etc.). The control unit 12 then saves the processed state according to this object (step S304).

[0098] If the recognition result has not changed since the previous time (No at step S302), the control unit 12 repeats the process from step S302 for the next row. Then, when all rows stored in the recognition interpretation information DB 11d have been processed, the control unit 12 ends the process at the determination processing unit 12c.

[0099] In the case of the Catan content, in step S701, the control unit 12 determines whether the detected object is placed at an intersection of hexagonal panels on the Catan content board, for example. This is because if the piece newly placed by the user is recognized as a settlement piece or a city piece, these pieces can only be placed at intersections on the board.

[0100] If the object is not placed at an intersection (No in step S701), the control unit 12 determines whether the score of the accuracy of the recognition result is equal to or greater than 80 points, which is the threshold defined in the score definition information 11b, in order to determine whether to display a dialog indicating that the object is placed in the wrong place (step S702). If the score is equal to or greater than 80 points (Yes in step S702), the processing in steps S703 to S705 is repeated until the condition of less than 80 points is met.

[0101] In step S703, the control unit 12 displays a warning (dialog) to the user on the board informing them that the placement is incorrect. Then, in step S704, the control unit 12 determines whether a certain time, which is a predefined presentation time for the dialog, has elapsed. If the time has not elapsed (step S704, No), step S704 is repeated. If the time has elapsed (step S704, Yes), step S705 is executed.

[0102] In step S705, the control unit 12 deducts 10 points, which is the correction dialog ignorance point deduction value defined in the score definition information 11b, from the accuracy of the recognition result because the correction dialog (warning dialog) indicating the incorrect placement was ignored. Also, because the user ignored the correction dialog, the control unit 12 adds 10 seconds, defined in the lock seconds definition information 11c, to the lock seconds.

[0103] These processes take the fact that the user has ignored the system's correction dialog as a starting point and deduct points from the accuracy of the recognition result, or extend the time until the recognition process is performed again, thereby preventing the user from being prompted to make multiple corrections due to incorrect recognition and disrupting the game's progress.

[0104] If the score is less than 80 points (step S702, Yes), the piece newly placed by the user is determined to be a "highway," and the interpretation result of the recognition interpretation information is changed to "highway" (step S706). Note that in the Catan content, road pieces are very similar in shape to settlement pieces and city pieces, but road pieces are placed on the sides of the board, not at intersections.

[0105] On the other hand, if the newly placed piece by the user is placed at an intersection (step S701, Yes), the control unit 12 determines whether the piece with the highest accuracy score among the recognition results may still be placed on the board (step S801). In the Catan content, a maximum of five settlement pieces and a maximum of four city pieces may be placed on the board.

[0106] If the piece should not exist (No in step S801), the number of pieces placed by the user exceeds the number that can exist on the board, and therefore 10 points, which is the over-number deduction value defined in the score definition information 11b, is deducted from the accuracy of the recognition result (step S808).Then, the control unit 12 repeats the process from step S701.

[0107] Also, if it is acceptable for it to exist (step S801, Yes), the control unit 12 determines whether or not to present a construction effect with audio by determining whether or not the accuracy of the recognition result is equal to or greater than 85 points, which is the threshold defined in the scoring definition information 11b (step S802).

[0108] If the score is 85 or more (step S802, Yes), the control unit 12 presents the construction effect with audio to the user (step S803). If the score is less than 85 (step S802, No), the control unit 12 presents the construction effect without audio to the user (step S806). In this way, by changing the presentation method according to the recognition result, it is possible to prevent the user's time from being wasted by erroneous and excessive presentation effects, and improve usability.

[0109] Then, the control unit 12 determines whether the user has requested a correction to the displayed construction effect in order to determine whether the piece placed by the user has been correctly recognized and the construction effect has been displayed (step S804).

[0110] If so (step S804, Yes), the control unit 12 determines that the piece placed by the user was not correctly recognized, and performs high-precision recognition (advanced recognition processing), which takes time, and adds 20 seconds defined in the lock seconds definition information 11c to the lock seconds (step S805). Because the recognition results obtained by the advanced recognition processing are more reliable than those obtained by the normal recognition processing, the control unit 12 does not perform the recognition processing again for a while.

[0111] Then, the control unit 12 executes step S802 for the recognition result of the advanced recognition process. If the user does not request correction in step S804 (step S804, No), the control unit 12 determines that the piece placed by the user has been correctly recognized, and does not change the recognition interpretation information (step S807).

[0112] Then, the control unit 12 saves the state after steps S706, S805, and S807 are executed (step S304).

[0113] <<4. Examples in Monopoly Content>> More specific examples executed by the information processing device 10 according to this embodiment will be described below. First, first to seventh examples, which are examples in Monopoly content, will be described with reference to FIGS. 10 to 23.

[0114] <4-1. First Example> Fig. 10 is an explanatory diagram of a first example. Fig. 11 is a diagram showing recognition interpretation information in the first example. This first example is an example in which a player's error is pointed out when the accuracy of the recognition result of the information processing device 10 is high.

[0115] As shown in FIG. 10, it is assumed that a player rolls dice, and the first die shows a 2 and the second die shows a 3, resulting in a total of 5 (step S10-1).

[0116] In response to this, the information processing device 10 erroneously recognizes that the first die is a 3, the second die is a 3, and the total is a 6, as shown in the recognition results in the upper table of Fig. 11. In this case, as shown in Fig. 10, the information processing device 10 displays a destination effect instructing the piece to move six squares forward (step S10-2).

[0117] On the other hand, since the information processing device 10 has made an incorrect recognition, the player ignores the destination effect and moves the piece five spaces ahead (step S10-3).

[0118] In this case, as shown in the table below in Fig. 11, the score (accuracy) of the dice recognition result is 95 points. This score exceeds the threshold of 90 points defined in the score definition information 11b. In this case, the information processing device 10 displays a dialog on the board stating that "the piece is placed in the wrong place" (step S10-4).

[0119] If there is no reaction from the player for a certain period of time or more after the dialog is displayed, the information processing device 10 determines that its own recognition result is incorrect. Then, as shown in the table below in FIG. 11 , the information processing device 10 adds 10, which is the defined value of the lock seconds defined in the lock seconds definition information 11c, to the lock seconds, and corrects the interpretation result so that the total value becomes 5. The information processing device 10 also corrects the position of the destination effect to the correct position (step S10-5).

[0120] Note that the table at the bottom of Figure 11 shows an example in which the information processing device 10 changes the interpretation result and locking seconds for the dice in the first row, but as long as the total value is consistent and equals 5, it is also possible to change the interpretation result and locking seconds for the dice in the second row instead of the first row, for example.

[0121] In this way, when the player ignores the information presented by the information processing device 10, the information processing device 10 can determine that the information it recognized was incorrect and correct the interpretation result, thereby preventing the frequent display of dialogs urging corrections.

[0122] 4-2. Second Example Next, Fig. 12 is an explanatory diagram of a second example. Fig. 13 is a diagram showing recognition interpretation information in the second example. Unlike the first example, this second example is an example in which a player's error is not pointed out when the accuracy of the recognition result of the information processing device 10 is low.

[0123] As shown in FIG. 12, it is assumed that a player rolls dice, and the first die shows a 2 and the second die shows a 3, resulting in a total of 5 (step S12-1).

[0124] 13, the information processing device 10 erroneously recognizes that the first die is a 3, the second die is a 3, and the total is a 6. In this case, as shown in FIG. 12, the information processing device 10 displays a destination effect instructing the player to move the piece six squares forward (step S12-2).

[0125] On the other hand, since the information processing device 10 has made an incorrect recognition, the player ignores the destination effect and moves the piece forward five spaces (step S12-3).

[0126] Here, according to the recognition result in Fig. 13, the score of the recognition result for the dice is 85 points, which is below the threshold of 90 points defined in the score definition information 11b. In this case, the information processing device 10 does not display a dialog box saying "the piece placement is incorrect," but corrects the interpretation result so that the total value becomes 5, as shown in Fig. 13. The information processing device 10 also corrects the position of the destination effect to the correct position (step S12-4).

[0127] In this way, when the accuracy of the recognition result is low, the information processing device 10 does not display a dialog box that unnecessarily prompts the user to make corrections, thereby preventing the progress of the game from being interrupted.

[0128] 4-3. Third Example Next, Fig. 14 is an explanatory diagram of a third example. Fig. 15 is a diagram showing recognition interpretation information in the third example. This third example is an example in which an effect is displayed when the accuracy of the recognition result of the information processing device 10 is high.

[0129] As shown in Fig. 14, assume that the player has built three houses on a square (step S14-1). Then, as shown in the recognition results in the upper table of Fig. 15, the information processing device 10 erroneously recognizes that two houses have been built, but the score is 95 points, which exceeds the threshold of 90 points defined in the score definition information 11b. In this case, the information processing device 10 displays an effect on the board indicating that two houses have been built (step S14-2).

[0130] In response to this, it is assumed that the player notices the erroneous recognition by the information processing device 10 and makes a corrective gesture, for example, by waving his / her hand (step S14-3).

[0131] When the information processing device 10 recognizes this correction gesture, it displays a correction dialogue that prompts the player to specify the correct number of houses with a gesture (step S14-4).The player specifies the correct number of houses with a gesture in accordance with this correction dialogue (step S14-5).

[0132] Then, when the information processing device 10 recognizes the correct number of houses from this gesture, it corrects the interpretation result to the content specified by the player (i.e., three houses) as shown in the lower table of Fig. 15. Then, the information processing device 10 displays an effect with the correct number of houses after the correction (step S14-6).

[0133] In this way, the information processing device 10 can correct the recognition result of its own incorrect recognition based on a simple gesture for correction by the player.

[0134] 4-4. Fourth Example Next, Fig. 16 is an explanatory diagram of a fourth example. Fig. 17 is a diagram showing recognition interpretation information in the fourth example. Unlike the third example, this fourth example is an example in which an effect is not displayed when the accuracy of the recognition result of the information processing device 10 is low.

[0135] 16, it is assumed that the player has built three houses in the square (step S16-1). Then, the information processing device 10 erroneously recognizes this action of the player as having built two houses, as shown in the recognition results for two rows in FIG.

[0136] 17, the score of the recognition result is 85 points, which is below the threshold of 90 points defined in the score definition information 11b. In this case, the information processing device 10 does not display an effect indicating that two houses have been built, but displays a correction dialog box that prompts the player to specify the correct number of houses with gestures (step S16-2).

[0137] When the player specifies the correct number of houses by gestures in accordance with the correction dialog (step S16-3), the information processing device 10 recognizes this and corrects the interpretation result to the content specified by the player (i.e., three houses), as shown in Fig. 17. The information processing device 10 then displays an effect with the correct number of houses after the correction (step S16-4).

[0138] In this way, when the accuracy of the recognition result is low, the information processing device 10 does not unnecessarily display effects, but asks the player for the correct result, thereby allowing the game to proceed smoothly.

[0139] 4-5. Fifth Example Next, Fig. 18 is an explanatory diagram of a fifth example. Fig. 19 is a diagram showing recognition interpretation information in the fifth example. This fifth example is an example in which, when the accuracy of the recognition result of the information processing device 10 is high, a warning is displayed indicating that placement is prohibited.

[0140] 18, it is assumed that the player places a house piece on a square where a house should not be placed (step S18-1).Then, the information processing device 10 recognizes this action of the player as a score of 95 points, as shown in the recognition results in the upper table of FIG.19 (step S18-2).

[0141] Even after subtracting 10 points, which is the deduction value for unplaceable squares defined in the score definition information 11b, this 95 points still exceeds 80 points, which is the threshold for presenting the misplaced box dialog, also defined in the score definition information 11b.

[0142] In this case, the information processing device 10 displays a dialogue on the board indicating that the house should not be placed on this square (step S18-3).

[0143] This allows the player to realize the mistake and remove the house piece from the corresponding square (step S18-4). In addition, the information processing device 10 deletes the recognition result, interpretation result, and lock time for the corresponding area, as shown in the table below in FIG.

[0144] In this way, if the accuracy of the recognition result is high and the accuracy of the recognition result is confirmed by exceeding the misplacement dialog presentation threshold even after deducting the 10-point penalty point for unplaceable squares, the information processing device 10 displays a warning dialog to the player, thereby allowing the player to correct their mistake and progress smoothly through the game.

[0145] 4-6. Sixth Example Next, Fig. 20 is an explanatory diagram of a sixth example. Fig. 21 is a diagram showing recognition interpretation information in the sixth example. In this sixth example, if the accuracy of the recognition result of the information processing device 10 is high, a warning is displayed indicating that placement is prohibited, but unlike the fifth example, the placement is accepted after a certain period of time has passed.

[0146] 20, it is assumed that the player places a house piece on a square where a house should not be placed (step S20-1).Then, the information processing device 10 recognizes this action of the player as a score of 95 points, as shown in the recognition results in the upper table of FIG.21 (step S20-2).

[0147] Even after subtracting the 10 points that are the deduction value for unplaceable squares, this score of 95 still exceeds the 80 points that is the threshold for displaying the misplaced box dialogue.

[0148] In this case, the information processing device 10 displays a dialogue on the board indicating that the house should not be placed on this square (step S20-3).

[0149] In response to this, it is assumed that the player ignores the dialogue and leaves the house for a certain period of time (step S20-4), in which case the information processing device 10 accepts the placement of the house placed by the player as the correct result.

[0150] Since building a house on this square has no effect, the information processing device 10 continues the game with the interpretation result set to "none," meaning that the dialogue will be ignored in the course of the game, as shown in the table at the bottom of Fig. 21. Furthermore, in order to retain information that the player has ignored the dialogue, the information processing device 10 adds 10 seconds to the lock seconds when "the dialogue is ignored," as defined in the lock seconds definition information 11c, as shown in the table at the bottom of Fig. 21.

[0151] This allows the information processing device 10 to prioritize the progress of the game while preventing warning dialogs from being displayed frequently.

[0152] 4-7. Seventh Example Next, Fig. 22 is an explanatory diagram of a seventh example. Fig. 23 is a diagram showing recognition interpretation information in the seventh example. Unlike the fifth and sixth examples, the seventh example is an example in which a warning is not displayed when the accuracy of the recognition result of the information processing device 10 is low.

[0153] As shown in Fig. 22, it is assumed that the player places a house piece on a square where a house should not be placed (step S22-1).Then, it is assumed that the information processing device 10 recognizes this action of the player as a score of 82 points, as also shown in Fig. 23 (step S22-2).

[0154] Note that, if the 10 points that are the deduction value for unplaceable squares are subtracted, this 82 points falls below the 80 points that are the threshold for presenting the misplaced box dialogue.

[0155] In this case, the information processing device 10 determines that there is a high possibility of erroneous recognition, and does not display a dialogue indicating that the square is one where a house should not be placed, and allows the game to continue as is (step S22-3).

[0156] In this way, when the accuracy of the recognition result is low, the information processing device 10 does not unnecessarily display a warning dialog, thereby preventing the progress of the game from being interrupted.

[0157] <<5. Examples of Catan Contents>> Next, eighth to twelfth examples, which are examples of Catan contents, will be described with reference to FIGS. 24 to 33. FIG.

[0158] 5-1. Eighth Example Fig. 24 is an explanatory diagram of an eighth example. Fig. 25 is a diagram showing recognition interpretation information in the eighth example. This eighth example is an example in which a warning is displayed when the accuracy of the recognition result of the information processing device 10 is high.

[0159] As shown in Figure 24, assume that a player places a settlement piece on an edge where settlement pieces are not allowed (step S24-1). In Catan, settlement and city pieces cannot be placed on the edges of hexagonal panels, but must be placed at the intersections of the panels.

[0160] 25, the information processing device 10 recognizes this action of the player as a score of 85 points. Note that this 85 points exceeds the misplaced object dialog presentation threshold of 80 points defined in the score definition information 11b.

[0161] In this case, the information processing device 10 displays a dialogue on the board to inform the player that the location is one where a settlement cannot be placed (step S24-2).

[0162] In response to this, it is assumed that the player ignores the dialogue and leaves the settlement alone for a certain period of time or more. In this case, the information processing device 10 deducts 10 points, which is the correction dialogue ignorance point deduction value defined in the score allocation definition information 11b, from the score of the recognition result shown in Fig. 25.

[0163] Since the result is below the misplaced area dialog presentation threshold of 80 points, the information processing device 10 stops displaying the dialog (step S24-3). Then, the information processing device 10 corrects the interpretation result so that the piece placed by the player is interpreted as a highway rather than a settlement, as shown in Figure 25. Furthermore, in order to retain the information that the player ignored the dialog, the information processing device 10 adds 10 seconds for the case where the "dialog is ignored" defined in the lock seconds definition information 11c to the lock seconds, as shown in Figure 25, to make the total number of lock seconds 20.

[0164] In this way, the information processing device 10 can proceed with the game without repeatedly presenting the correctness of the recognition result on the information processing device 10 side to the player when the player ignores the dialogue.

[0165] In this example, a settlement has actually been mistakenly placed on an edge, so the recognition result of the information processing device 10 is not incorrect. However, since it is possible that the player mistakenly used a settlement piece as a road piece, the information processing device 10 takes this into consideration and corrects the interpretation result to a road. This is because, although there is no road next to the recognition result, the rules of Catan dictate that only road pieces can be placed on edges.

[0166] 5-2. Ninth Example Next, Fig. 26 is an explanatory diagram of a ninth example. Fig. 27 is a diagram showing recognition interpretation information in the ninth example. Unlike the eighth embodiment, this ninth example is an example in which a warning is not displayed when the accuracy of the recognition result of the information processing device 10 is low.

[0167] As shown in FIG. 26, it is assumed that a player places a settlement piece on an edge where a settlement piece should not be placed (step S26-1).

[0168] 27, the information processing device 10 recognizes this action of the player as a score of 75 points. Note that this 75 points is below the misplaced object dialog presentation threshold of 80 points.

[0169] In this case, the information processing device 10 does not display a dialogue indicating that the location is not permitted for placing a settlement, and allows the game to continue as is (step S26-3). Note that although the information processing device 10 does not display anything here, because the settlement piece is placed on an edge, it is possible to not impede the progress of the game by interpreting it as a highway, as shown in the interpretation result of Figure 27.

[0170] In this way, when the accuracy of the recognition result is low, the information processing device 10 can avoid interrupting the progress of the game by not unnecessarily displaying a warning dialog.

[0171] 5-3. Tenth Example Next, Fig. 28 is an explanatory diagram of a tenth example. Fig. 29 is a diagram showing recognition interpretation information in the tenth example. This tenth example is an example of executing advanced recognition processing with higher recognition performance than normal recognition processing.

[0172] As shown in Fig. 28, the player places a settlement piece (step S28-1), but the information processing device 10 mistakenly recognizes it as a city rather than a settlement, as shown in the recognition results in the upper table of Fig. 29. However, the score is 95 points as shown in Fig. 29, which exceeds the 85 points that is the threshold for presenting an effect with sound defined in the score definition information 11b.

[0173] Therefore, the information processing device 10 displays the city construction effect on the board with sound (step S28-2). Note that the musical note mark in the figure indicates that sound is included.

[0174] In response to this, it is assumed that the player notices the erroneous recognition by the information processing device 10 and makes a corrective gesture, for example, by waving his / her hand (step S28-3).

[0175] When the information processing device 10 recognizes this correction gesture, it executes an advanced recognition process that takes more time than normal recognition processing but has higher recognition performance (step S28-4).

[0176] As a result of the advanced recognition processing, the information processing device 10 recognizes the area as a settlement with a score of 95, as shown in the recognition results in the table below in Fig. 29. This score exceeds the aforementioned threshold of 85 for presenting audio-accompanying effects.

[0177] Therefore, the information processing device 10 displays a settlement construction effect on the board with audio (step S28-5). Furthermore, in order to retain information that high-precision recognition processing has been performed, the information processing device 10 adds 20 seconds for "performing advanced recognition processing" defined in the locking seconds definition information 11c to the locking seconds, as shown in the table below in Figure 29. This 20 seconds for "performing advanced recognition processing" is longer than the 10 seconds for "performing normal recognition processing" also defined in the locking seconds definition information 11c.

[0178] In this way, the information processing device 10 can correct the recognition result without the player being explicitly instructed to correct it by performing advanced recognition processing in response to the player's correction gesture. Note that because advanced recognition processing takes time, it is preferable to perform it as needed as shown in this embodiment, rather than performing it all the time.

[0179] 5-4. Eleventh Example Next, Fig. 30 is an explanatory diagram of an eleventh example. Fig. 31 is a diagram showing recognition interpretation information in the eleventh example. This eleventh example is an example in which not only the recognition results of a single piece are taken into consideration, but also the number of pieces already placed on the board, etc.

[0180] As shown in Figure 30, the player places a city piece (step S30-1), and as shown in the recognition result in Figure 31, the information processing device 10 recognizes it as a settlement with a score of 85, or as a city with a score of 80.

[0181] In this case, the information processing device 10 interprets that the new piece placed by the player belongs to the settlement with the higher score, based on the size of the score. Here, as shown in step S30-2 of Figure 30, it is assumed that four settlement pieces have already been placed on the board (see (1) to (4) in the figure).

[0182] According to the rules of Catan, only four pieces can be placed on a settlement. Therefore, the information processing device 10 deducts 10 points for any settlement that the player interprets as being newly placed, which is the over-count deduction value defined in the score definition information 11b.

[0183] As a result of the point deduction, the score of the settlement becomes 75 points, which is lower than the city's score of 80 points. As a result, the information processing device 10 reinterprets the piece newly placed by the player as a city, as shown in the interpretation result of FIG.

[0184] However, the score of the city is 80 points, which is below the threshold of 85 points for presenting an effect with sound defined in the score definition information 11b. Therefore, the information processing device 10 displays the city construction effect on the board without sound (step S30-3).

[0185] In this way, the information processing device 10 can derive correct recognition results by taking into consideration not only the recognition results of each individual piece, but also how many pieces are already present on the board and how many pieces are allowed to be present according to the rules.

[0186] 5-5. Twelfth Example Next, Fig. 32 is an explanatory diagram of a twelfth example. Fig. 33 is a diagram showing recognition interpretation information in the twelfth example. This twelfth example is an example of a case where an effect is changed depending on the score of the interpretation result.

[0187] As shown in FIG. 32, the player places a settlement piece (step S32-1), and as shown in the recognition result in FIG. 33, the information processing device 10 recognizes it as a settlement with a score of 80 points.

[0188] However, the score of the settlement is below the threshold of 85 points for presenting an effect with sound defined in the score definition information 11b, so the information processing device 10 displays the settlement construction effect on the board without sound (step S32-2).

[0189] In this way, the information processing device 10 can selectively use a simple effect without sound for a short time or a flashy effect with sound for a long time depending on the score of the recognition result. This allows the information processing device 10 to prevent the player from wasting time with flashy effects due to the erroneous recognition, even if the recognition result is incorrect.

[0190] The different effects may be used depending on the media, such as with or without audio, or may be used depending on the presentation time of the effect, such as long or short.

[0191] <<6. Examples of Various Corrections in Game Content>> Next, examples 13 to 22, which are examples of various corrections in game content, will be described with reference to Figures 34 to 43. Note that the case of Monopoly content will be used as an example.

[0192] In the following, player A will be the player located at the bottom of the board in a plan view of the board in each of the figures shown below. Similarly, player B will be the player located to the right of the board. Similarly, player C will be the player located at the top of the board. Similarly, player D will be the player located to the left of the board.

[0193] 6-1. Thirteenth Example Fig. 34 is an explanatory diagram of a thirteenth example. In this thirteenth example, the player who can make corrections is limited to the player himself, and only the player himself can make corrections.

[0194] 34, it is assumed that player A rolls the dice (step S34-1). Furthermore, it is assumed that the information processing device 10 erroneously recognizes the total value of the dice as 7, even though it is 9, and displays a destination effect 7 squares ahead (step S34-2).

[0195] It is assumed that at this time, it is Player A who rolled the dice who notices the misrecognition by the information processing device 10 and makes a corrective gesture (step S34-3).

[0196] Then, the information processing device 10 displays a correction GUI (Graphical User Interface) in Player A's hand, which allows the player to specify the correct total value of the dice (step S34-4).

[0197] Then, when the information processing device 10 recognizes that a total value has been specified by player A via this correction GUI, it corrects the display content on the board, for example, by changing the position of the destination effect based on the specified total value (step S34-5).

[0198] In this way, the information processing device 10 allows only authorized players to make corrections, and determines whether the person making the correction gesture is, for example, the person who rolled the dice, and allows the correction only if that person is the authorized player.

[0199] 35 is an explanatory diagram of a fourteenth embodiment. In this fourteenth embodiment, as in the thirteenth embodiment, only the player who can make corrections is allowed to make corrections, but this is an example in which a third party may attempt to make a correction.

[0200] 35, it is assumed that player A rolls the dice (step S35-1). Furthermore, it is assumed that the information processing device 10 erroneously recognizes the total value of the dice as 7, even though it is 9, and displays a destination effect 7 squares ahead (step S35-2).

[0201] At this time, it is assumed that it is player B, not player A who rolled the dice, who notices the misrecognition by the information processing device 10 and makes a corrective gesture (step S35-3).

[0202] Then, the information processing device 10 displays a dialog box to Player B to inform him that he does not have the authority to make corrections (step S35-4).

[0203] In this way, the information processing device 10 determines whether the person making the correction gesture is, for example, the person who rolled the dice, and can prohibit the correction if only authorized players are allowed to make the correction.

[0204] 36 is an explanatory diagram of a fifteenth embodiment. In this fifteenth embodiment, the players who can make corrections are limited to only the parties who are interacting in the game, and only the parties can make corrections.

[0205] As shown in FIG. 36, as the game progresses, player C hands over $250 in game bills to player A (step S36-1), and player A receives them (step S36-2).

[0206] However, the information processing device 10 mistakenly recognizes the amount exchanged as $150 and displays the balance amounts for each of Player A and Player C (step S36-3).

[0207] It is assumed that it is Player A who notices the misrecognition by the information processing device 10 and makes a corrective gesture at this time (step S36-4).

[0208] Then, the information processing device 10 displays a correction GUI in Player A's hand that allows the player A to specify the correct amount exchanged (step S36-5).

[0209] Then, when the information processing device 10 recognizes that an amount has been specified by player A via this correction GUI, it displays a GUI in the hand of player C, a party other than player A, that allows the player C to approve or disapprove the amount specified by player A (step S36-6).

[0210] Then, when the information processing device 10 recognizes that Player C has approved, it corrects the initial recognition result of $150 in accordance with the approved amount.

[0211] In this way, the information processing device 10 determines that only authorized parties can make corrections, and determines whether the person who made the correction gesture is, for example, a party who exchanged game banknotes, and allows corrections only if the person is an authorized party.

[0212] 37 is an explanatory diagram of Example 16. As with Example 15, Example 16 limits the players who can make corrections to only those who are involved in the game, but it is an example in which a player other than the involved players attempts to make a correction.

[0213] As shown in FIG. 37, as the game progresses, player C hands over $250 in game bills to player A (step S37-1), and player A receives them (step S37-2).

[0214] However, the information processing device 10 mistakenly recognizes the amount exchanged as $150 and displays the balance amounts for each of Player A and Player C (step S37-3).

[0215] It is assumed that it is player B, who is not the party involved, who notices the misrecognition by the information processing device 10 and makes a corrective gesture (step S37-4).

[0216] Then, the information processing device 10 displays a dialog box to Player B to inform him that he does not have the authority to make corrections (step S37-5).

[0217] In this way, the information processing device 10 determines whether the person who made the correction gesture is, for example, the party who exchanged the game banknote, and determines that only authorized parties can make corrections, and if the person is not an authorized party, the correction is not allowed.

[0218] 6-5. Seventeenth Example Next, Fig. 38 is an explanatory diagram of a seventeenth example. This seventeenth example is an example in which confirmation is sought from other players when making corrections.

[0219] 38, it is assumed that player A rolls the dice (step S38-1). Furthermore, it is assumed that the information processing device 10 erroneously recognizes the total value of the dice as 7, even though it is 9, and displays a destination effect seven squares ahead (step S38-2).

[0220] However, the property seven spaces away has been purchased by Player B, and according to the rules, if Player A were to land on that space, he would have to pay a large penalty to Player B.

[0221] Then, it is assumed that Player A realizes the misrecognition by the information processing device 10 and makes a corrective gesture (step S38-3).

[0222] Then, the information processing device 10 displays a correction GUI in Player A's hand that allows the player A to specify the correct total value of the dice (step S38-4).

[0223] Then, when the information processing device 10 recognizes that a total value has been specified by player A via this correction GUI, it displays a GUI to each player other than player A, requesting confirmation of the total value specified by player A (step S38-5).

[0224] At this time, the information processing device 10 displays a GUI that prompts the player to make an approval gesture, such as "clap their hands," if they approve. Note that player B may choose not to make an approval gesture because he or she would not receive the high penalty that would be obtained if he or she approved.

[0225] Therefore, when the information processing device 10 recognizes that players C and D have made approval gestures, it approves the total value designated by player A by majority vote (step S38-6).

[0226] In this way, the information processing device 10 can prevent the game from proceeding unfairly due to, for example, selfish behavior by some players, by checking with other players whether a correction operation by a certain player is appropriate.

[0227] 6-6. Eighteenth Example Next, Fig. 39 is an explanatory diagram of an eighteenth example. This eighteenth example is an example in which corrections from the player are accepted only within a certain period of time.

[0228] 39, it is assumed that player A rolls the dice (step S39-1). Furthermore, it is assumed that the information processing device 10 erroneously recognizes the total value of the dice as 7, even though it is 9, and displays a destination effect 7 squares ahead (step S39-2).

[0229] Then, it is assumed that at this time, Player A notices the erroneous recognition by the information processing device 10 and makes a corrective gesture (step S39-3).

[0230] Then, if this correction gesture is made within a certain period of time (e.g., 5 seconds) after the display of the destination effect, the information processing device 10 displays a correction GUI in Player A's hand that allows the player to specify the correct total value of the dice (step S39-4).

[0231] Then, when the information processing device 10 recognizes that a total value has been specified by player A via this correction GUI, it corrects the initial recognition result of 7 based on the specified total value. On the other hand, if the correction gesture is made after a certain period of time has elapsed, the information processing device 10 does not accept the correction.

[0232] In this way, the information processing device 10 accepts correction gestures within a certain period of time, but does not accept them if more time has passed, thereby preventing corrections from being made to the game progress going back too far.

[0233] 6-7. 19th Example Next, Fig. 40 is an explanatory diagram of a 19th example. This 19th example is an example in which corrections are not accepted once the player has given permission.

[0234] 40, it is assumed that player A rolls the dice (step S40-1). Furthermore, it is assumed that the information processing device 10 erroneously recognizes the total value of the dice as 7, even though it is 9, and displays a destination effect 7 squares ahead (step S40-2).

[0235] It is assumed that Player A moves his / her piece seven spaces ahead without noticing the misrecognition of the information processing device 10 (step S40-3). Then, Player A realizes that he / she must pay, for example, a large amount of game bills seven spaces ahead, and upon closer inspection, he / she sees that the total value of the dice is 9, so he / she realizes the misrecognition of the information processing device 10 and makes a corrective gesture (step S40-4).

[0236] In this case, the information processing device 10 displays a dialog box to Player A to inform him that the correction cannot be accepted because Player A has already moved the piece once (step S40-5).

[0237] In this way, the information processing device 10 considers that the player has accepted the recognition result of the information processing device 10 once the player has moved a piece, and does not accept any further corrections, thereby preventing the progress of the game from being corrected going back significantly.

[0238] 41 is an explanatory diagram of a twentieth embodiment. This twentieth embodiment is an example in which, even if there is an erroneous recognition by the information processing device 10, once another player has started an operation, the correction is not accepted.

[0239] 41, it is assumed that player A rolls the dice (step S41-1). Furthermore, it is assumed that the information processing device 10 erroneously recognizes the total value of the dice as 7, even though it is 9, and displays a destination effect seven squares ahead (step S41-2).

[0240] It is assumed that player A does not notice the misrecognition of the information processing device 10 and moves his piece seven spaces ahead (step S41-3). It is then player B's turn, and player B builds a house (step S41-4).

[0241] It is assumed that player A now realizes the misrecognition by the information processing device 10 and makes a corrective gesture (step S41-5).

[0242] In this case, the information processing device 10 displays a dialog box to Player A to inform him that the correction is not acceptable because the game has already moved to Player B's turn (step S41-6).

[0243] In this way, the information processing device 10 can prevent the progress of the game from being corrected significantly back in time by not accepting any corrections made by the previous player once the game has moved on to another player's turn, without having to follow the seventh embodiment described above.

[0244] 42 is an explanatory diagram of Example 21. Example 21 is an example in which only operations in the vicinity of the object to be corrected are regarded as correction gestures.

[0245] 42, it is assumed that player A rolls the dice (step S42-1). Furthermore, it is assumed that the information processing device 10 erroneously recognizes the total value of the dice as 7, even though it is 9, and displays a destination effect 7 squares ahead (step S42-2).

[0246] Now, assume that player B happens to make a movement on the board that resembles a correction gesture due to habit or the like (step S42-3). In this case, the information processing device 10 ignores anything outside a predetermined range (see the dashed closed curve in the figure) centered on the die to be corrected, as it is outside the range of recognition for a correction gesture. Therefore, the information processing device 10 does not recognize player B's hand movement as a correction gesture.

[0247] On the other hand, if Player A makes a correction gesture near the dice (step S42-4), the information processing device 10 recognizes this and displays a correction GUI near Player A's hand that enables the player to specify the correct total value of the dice (step S42-5).

[0248] In this way, the information processing device 10 can prevent unintentional corrections by limiting the area in which a correction gesture by the player is accepted to the vicinity of the object to be corrected.

[0249] 6-10. 22nd Example Next, Fig. 43 is an explanatory diagram of a 22nd example. This 22nd example is an example of a correction operation.

[0250] Up to now, an example has been given in which the correction operation is a "gesture" of waving a hand as shown in the left diagram of Fig. 43, but the correction operation is not limited to this example. For example, the information processing device 10 may display a "GUI" for correction as shown in the middle diagram of Fig. 43, and may treat a tap operation by the player on this as a correction operation.

[0251] In the case of "GUI," for example, after a player rolls the dice, the information processing device 10 displays a correction GUI for this correction on the board or in the player's hand for a certain period of time.

[0252] Furthermore, as shown in the right diagram of Figure 43, the correction operation may involve the player holding up a "correction card" for correction. These correction operations may be used independently or in combination.

[0253] Furthermore, since "GUI" and "correction card" occur less frequently by chance than "gestures" such as waving a hand, if correction operations are recognized at the same time, the information processing device 10 may accept the correction operations by prioritizing, for example, "correction card," "GUI," and "gesture" in that order.

[0254] <<7. Examples Related to Object Recognition>> Next, the 23rd to 30th examples related to object recognition will be described with reference to FIGS. 44 to 51. FIG.

[0255] 7-1. 23rd Example> Fig. 44 is an explanatory diagram of the 23rd example. The 23rd example is an example in which ease / difficulty of recognition is tested in advance.

[0256] As shown in FIG. 44, the information processing device 10 can roll digital dice, which are virtual dice, and conduct a dice recognition test before starting a game (step S44-1).

[0257] Then, the information processing device 10 performs the dice recognition test several times to find a location where the accuracy of recognition is low and where misrecognition is likely to occur (step S44-2). Here, for example, the upper right corner of the table where the light is reflecting is assumed to be the location. Note that the information processing device 10 rolls virtual digital dice instead of actual dice, so it can find the location by comparing the result with the correct answer.

[0258] Then, the information processing device 10 tries to correct various shooting parameters such as brightness, and finds a correction method that allows correct recognition (step S44-3). Here, it is assumed that correct recognition is achieved by performing the recognition process with the exposure time of the camera of the sensor unit 3 reduced, for example.

[0259] The information processing device 10 then stores this correction method and performs recognition processing while applying this individuality method during game play (step S44-4). As shown in the figure, the correction method may include, for example, brightness correction, color correction, and processing correction such as sharpening and contrast.

[0260] In this way, the information processing device 10 can automatically determine the ease / difficulty of recognition of each part on the object recognition surface and automatically adjust the shooting parameters by, for example, performing a digital dice recognition process before the start of a game. Furthermore, the information processing device 10 can improve recognition accuracy and reduce erroneous recognition by dividing the recognition surface into specific areas and detecting the characteristics of each area (such as differences in brightness and color).

[0261] 45 is an explanatory diagram of a 24th embodiment. This 24th embodiment is an example of guiding an object so that it does not exist in an area where recognition accuracy is low.

[0262] As shown in FIG. 45, the information processing device 10 rolls the digital dice and performs a dice recognition test before starting the game, similarly to the 23rd embodiment (step S45-1).

[0263] Then, the information processing device 10 performs the dice recognition test several times to find a location where the accuracy of recognition is low and where erroneous recognition is likely to occur. Here, as in the 23rd embodiment, it is assumed that the location is the upper right corner of the table where the light is reflected.

[0264] However, unlike the twenty-third embodiment, it is assumed that the information processing device 10 has found that correct recognition is not possible even if the exposure time of the camera of the sensor unit 3 is reduced (step S45-2).

[0265] In this case, the information processing device 10 uses, for example, the relevant location as an area for displaying a fixed GUI (step S45-3).

[0266] If such a fixed GUI is displayed, it is expected that the player will avoid rolling the dice in that location. In other words, the information processing device 10 can guide the player not to roll the dice in that location while playing the game.

[0267] 46 is an explanatory diagram of a 25th embodiment. This 25th embodiment is an example in which areas with high recognition accuracy are clearly indicated.

[0268] As shown in FIG. 45, the information processing device 10 rolls the digital dice and performs a dice recognition test before starting the game, similarly to the 23rd and 24th examples.

[0269] Then, the information processing device 10 performs the dice recognition test several times to find a location where the accuracy of recognition is low and where erroneous recognition is likely to occur. Here, as in the 23rd and 24th examples, it is assumed that the location in question is, for example, the upper right corner of the table where the light is reflected.

[0270] The game is then started in this state, and the information processing device 10 again misrecognizes the numbers on the dice many times, resulting in frequent inconsistencies in the numbers on the dice (step S46-1).

[0271] In this case, the information processing device 10, for example, indicates an area on the board with high recognition accuracy, and displays a dialogue to the player asking him to move the dice to that area (step S46-2).

[0272] Then, the information processing device 10 can correctly recognize the number on the dice by having the player move the dice to the corresponding area.

[0273] 7-4. 26th Example Next, Fig. 47 is an explanatory diagram of a 26th example. This 26th example is an example of displaying an effect that encourages alignment.

[0274] 47, it is assumed that the player randomly places three house pieces on a grid (step S47-1), and as a result, the information processing device 10 mistakenly recognizes the number of houses and displays an effect that indicates that two houses have been constructed (step S47-2).

[0275] In response to this, it is assumed that the player makes a correction gesture (step S47-3). When the information processing device 10 recognizes this correction gesture, it displays a correction dialog that prompts the player to specify the correct number of houses with a gesture. In addition to this correction dialog, the information processing device 10 also displays guidance that encourages the player to align and rearrange the house pieces (step S47-4).

[0276] Then, the player follows this guidance to align and rearrange the house pieces (step S47-5), which allows the information processing device 10 to correctly recognize the number of houses (step S47-6).

[0277] By displaying an effect that encourages alignment in this way, the information processing device 10 can encourage the player to arrange the tiles in a way that is convenient for correct recognition.

[0278] 7-5. 27th Example Next, Fig. 48 is an explanatory diagram of the 27th example. This 27th example is an example of displaying a search effect. The search effect is, for example, an effect in which a searchlight partially illuminates the board surface.

[0279] As shown in FIG. 48, the information processing device 10 attempts to recognize the dice rolled by the player, but the projected image and the player's hand overlap the dice, and it is estimated that the recognition accuracy will be reduced (step S48-1).

[0280] In this case, the information processing device 10 displays the search effect as an animation approaching the player's hand (step S48-2), and also hides the image superimposed on the search effect (step S48-3).

[0281] This allows the information processing device 10 to improve the accuracy of dice recognition by causing the player's hand to move away from the board as the search effect approaches and by hiding the image that overlaps the search effect.

[0282] 49 is an explanatory diagram of the 28th embodiment. The 28th embodiment is an example (part 1) in which the player specifies the recognition target range within the capture target range of the camera of the sensor unit 3.

[0283] 49, the information processing device 10 can display a through image from the camera on a projection surface of the game board, such as the table top, before the start of a game (step S49-1). Note that in this case, it is assumed that the camera is photographing the table from diagonally above.

[0284] In this case, the player can fill in any part of the displayed through image that he or she does not want to be included in the recognition target range by touching the image with his or her finger (step S49-2).

[0285] Then, the information processing device 10 sets the area other than the filled-in portion as the recognition target range for the recognition process (step S49-3).

[0286] In this way, by limiting the recognition target range in the recognition process, the information processing device 10 can prevent unnecessary recognition processes from being performed on parts of the camera's capture target range that are not necessary for the progress of the game, thereby improving recognition accuracy.

[0287] 50 is an explanatory diagram of a 29th embodiment. This 29th embodiment is an example (part 2) in which the player specifies the recognition target range within the capture target range of the camera of the sensor unit 3.

[0288] As shown in FIG. 50, the camera takes an image from above of the tabletop, which is the projection surface of the game board (step S50-1).

[0289] In this case, the information processing device 10 can display a through image from this camera on the board (step S50-2).

[0290] In this case, the player can fill in the displayed through image by touching or otherwise manipulating the area whose state changes unrelated to the game while the game is in progress (step S50-3). Here, it is assumed that the player is filling in the area on the table where a basket of fruit is placed.

[0291] The information processing device 10 then sets the area other than the filled-in area as the recognition target range for the recognition process. Therefore, even if the amount of fruit in the basket decreases during the game, this does not affect the recognition process of the information processing device 10 (step S50-3).

[0292] In this way, the information processing device 10 can achieve stable recognition processing by excluding in advance from the recognition target range parts whose state changes unrelated to the game while the game is in progress.

[0293] In the twenty-eighth and twenty-ninth examples, examples have been given in which the player indirectly specifies the recognition target range by specifying a part outside the recognition target range, but the player may also directly specify the recognition target range by touch operation or the like.

[0294] 51 is an explanatory diagram of a 30th embodiment. The 30th embodiment is an example in which a recognition target range within a capture target range of the camera of the sensor unit 3 is automatically detected.

[0295] As shown in FIG. 51, the camera photographs the tabletop, which is the projection surface of the game board, from above, in the same manner as in the 29th embodiment (step S51-1).

[0296] Then, the information processing device 10 sets the center of the camera's through image as a starting point and expands the recognition target range to the surrounding area continuing along the planar direction from the starting point (step S51-2).

[0297] Then, the information processing device 10 detects the outer edge of the table that exists outside the board surface as an edge surrounding the board surface, and sets the area surrounded by this outer edge as the recognition target range (step S51-3).

[0298] Therefore, even if a cat or the like crosses the floor while the game is in progress, it does not affect the recognition process of the information processing device 10 (step S51-4).

[0299] In this way, the information processing device 10 can realize stable recognition processing by automatically excluding from the recognition target range parts whose state changes unrelated to the game while the game is in progress.

[0300] <<8. Examples Other Than Game Content>> Next, examples 31 to 37, which are examples other than game content, will be described with reference to FIGS. 52 to 58.

[0301] 52 is an explanatory diagram of the 31st embodiment. The 31st embodiment is an example in which questions outside the scope of previous learning are skipped in a learning application for educational purposes.

[0302] 52, when a user uses a learning application, the information processing device 10 displays a question and the page number of the textbook corresponding to the question on the projection screen (step S52-1). Here, it is assumed that a question outside the range of knowledge the user has already learned is displayed (step S52-2).

[0303] In this case, the user can make a correction gesture such as waving their hand, just as in the case of game content (step S52-3). If the information processing device 10 recognizes this correction gesture, it recognizes that the displayed question is outside the range of what the user has already learned, and displays a new question from a different learning range (step S52-4).

[0304] This allows the information processing device 10 to display questions appropriately depending on whether they are outside the range of knowledge the user has already acquired, thereby improving convenience when studying using a learning application.

[0305] Note that, regardless of whether it is within the learned range, when, for example, the user slides their hand to the left, the information processing apparatus 10 may display a new problem statement such that the page number of the textbook advances, and when the user slides their hand to the right, the page number of the textbook returns.

[0306] <8-2. 32nd Embodiment> Next, FIG. 53 is an explanatory diagram of the 32nd embodiment. This 32nd embodiment is an example in which re-recognition is performed when the stroke order of a Chinese character is misrecognized in a learning application for educational purposes.

[0307] As shown in FIG. 53, in a learning application in which the user learns the stroke order of Chinese characters, the information processing apparatus 10 recognizes one stroke of the Chinese character written by the user using a finger or the like (step S53-1).

[0308] However, here, it is assumed that the information processing apparatus 10 misrecognized, and one stroke of the Chinese character displayed by the information processing apparatus 10 as being written by the user was not what the user intended (step S53-2).

[0309] In this case, the user can perform a correction gesture such as waving their hand (step S53-3). Then, when the information processing apparatus 10 recognizes this correction gesture, it executes recognition processing again (for example, advanced recognition processing). As a result, the information processing apparatus 10 correctly recognizes that, for example, the stroke intended by the user is the vertical line on the left side rather than the right side inside the keibu (冂部) of the character "册" (step S53-4).

[0310] Thereby, in a learning application in which the information processing apparatus 10 learns the stroke order of Chinese characters, when a misrecognition unintended by the user occurs, the information processing apparatus 10 performs re-recognition based on the user's correction gesture and correctly recognizes it, enabling the user to correctly learn the stroke order.

[0311] <8-3. 33rd Embodiment> Next, FIG. 54 is an explanatory diagram of the 33rd embodiment. This 33rd embodiment is an example in which re-recognition is performed when a menu is misrecognized in a customer service application for restaurant use.

[0312] As shown in FIG. 54, in a customer service application for restaurant use, the information processing device 10 recognizes food placed on a table (step S54-1).

[0313] Here, it is assumed that the information processing device 10 has mistakenly recognized that the dish on the table is omurice but is a hamburger steak. In this case, the information processing device 10 displays an explanatory text about the hamburger steak around the dish (step S54-2).

[0314] It is assumed that the user then realizes this erroneous recognition and performs a corrective gesture such as waving their hand (step S54-3).When the information processing device 10 recognizes this corrective gesture, it executes the recognition process (for example, advanced recognition process) again.

[0315] As a result, the information processing device 10 recognizes that the dish on the table is omelet rice, and displays an explanatory text about omelet rice around the dish (step S54-4).

[0316] As a result, when the information processing device 10 misrecognizes a dish in a customer service application for restaurant use, it can re-recognize the dish correctly based on the user's correction gesture and display the correct description of the dish.

[0317] 55 is an explanatory diagram of a 34th embodiment. Similar to the 33rd embodiment, the 34th embodiment is an example in which, when a menu is erroneously recognized in a customer service application for restaurant use, a higher layer including the menu as one of its elements is displayed to prompt the user to correct the erroneous recognition.

[0318] As shown in FIG. 55, in a customer service application for restaurant use, the information processing device 10 recognizes the food placed on the table in the same manner as in the thirty-third embodiment (step S55-1).

[0319] Here, it is assumed that the information processing device 10 has mistakenly recognized that the dish on the table is omurice but is a hamburger steak. In this case, the information processing device 10 displays an explanatory text about the hamburger steak around the dish (step S55-2).

[0320] It is assumed that the user then realizes this misrecognition and performs a corrective gesture such as waving their hand (step S55-3). When the information processing device 10 recognizes this corrective gesture, it displays an upper layer including the misrecognized menu item as one of its elements around the dish, prompting the user to make a correction (step S55-4).

[0321] When the user selects, for example, omelet rice from each menu included in this upper layer, the information processing device 10 displays an explanatory text about the selected omelet rice around the dish.

[0322] In this way, when the information processing device 10 misrecognizes a dish in a customer service application for restaurant use, it displays a higher layer based on the user's correction gesture to prompt the user to make a correction, and then allows the user to select the correct menu item, thereby displaying the correct description of the dish.

[0323] Note that the information processing device 10 may be configured to display a higher layer (such as a main dish, salad, or drink) if the user performs a further correction gesture from the state of step S55-4. This allows correction even if a genre such as a main dish, salad, or drink is incorrectly recognized.

[0324] 56 is an explanatory diagram of a 35th embodiment. This 35th embodiment is an example in which, when an ingredient is erroneously recognized in an application for recording cooking procedures for cooking, re-recognition is performed.

[0325] As shown in FIG. 56, in an application for recording cooking procedures for cooking, the information processing device 10 recognizes ingredients that the user has put into a frying pan (step S56-1).

[0326] Here, it is assumed that the information processing device 10 mistakenly recognizes that the added ingredient is garlic but is actually onion. In this case, the information processing device 10 displays a cooking procedure in which onions are added around the periphery of the frying pan (step S56-2).

[0327] It is assumed that the user then realizes this erroneous recognition and performs a corrective gesture such as waving their hand (step S56-3).When the information processing device 10 recognizes this corrective gesture, it executes the recognition process (for example, advanced recognition process) again.

[0328] As a result, the information processing device 10 recognizes that the added ingredient is garlic, and displays a cooking procedure that includes putting garlic around the frying pan (step S56-4).

[0329] As a result, when the information processing device 10 misrecognizes ingredients in an application for recording cooking procedures for cooking purposes, it can re-recognize them correctly based on the user's correction gesture and display the correct cooking procedures.

[0330] 57 is an explanatory diagram of Example 36. Similar to Example 35, Example 36 is an example in which, when an ingredient is erroneously recognized in an application for recording cooking procedures for cooking, options are displayed, for example, on a cutting board.

[0331] As shown in FIG. 57, in an application for recording cooking procedures for cooking, the information processing device 10 recognizes ingredients that the user has cut on a cutting board (step S57-1).

[0332] Here, it is assumed that the information processing device 10 mistakenly recognizes that the cut ingredient is garlic but is actually an onion. In this case, the information processing device 10 displays a cooking procedure of cutting onions around the cutting board (step S57-2).

[0333] It is assumed that the user then realizes this misrecognition and performs a corrective gesture such as waving their hand (step S57-3). When the information processing device 10 recognizes this corrective gesture, it prompts the user to make a correction by displaying, for example, on a cutting board, one or more options for other ingredients that were recognized by the recognition process even though the recognition accuracy was lower than that of onion (step S57-4).

[0334] When the user selects, for example, garlic from these options, the information processing device 10 displays a cooking procedure for chopping garlic around the cutting board.

[0335] In this way, when the information processing device 10 misrecognizes ingredients in an application for recording cooking procedures for cooking, it displays other ingredient options based on the user's correction gesture and prompts the user to make a correction. Then, by having the user select the correct ingredients, it is possible to display the correct cooking procedures.

[0336] 58 is an explanatory diagram of Example 37. Similar to Examples 35 and 36, Example 37 is an example in which, when an ingredient is erroneously recognized in an application for recording cooking procedures for cooking, options that can be selected by gestures are displayed on a cutting board, for example.

[0337] As shown in FIG. 58, in an application for recording cooking procedures for cooking purposes, the information processing device 10 recognizes ingredients that the user has cut on a cutting board (step S58-1).

[0338] Here, it is assumed that the information processing device 10 mistakenly recognizes that the cut ingredient is garlic but is actually an onion. In this case, the information processing device 10 displays a cooking procedure of cutting onions around the cutting board (step S58-2).

[0339] It is assumed that the user then realizes this misrecognition and makes a corrective gesture such as waving their hand (step S58-3). When the information processing device 10 recognizes this corrective gesture, it prompts the user to make a correction by displaying, for example, on a cutting board, one or more options for other ingredients that were recognized by the recognition process even though the recognition accuracy was lower than that of onion.

[0340] At this time, the information processing device 10 displays, for example, options together with pictograms of gestures corresponding to the options. Then, when the user performs a gesture represented by the pictogram corresponding to the option for garlic, for example, the information processing device 10 displays a cooking procedure for chopping garlic around the periphery of a cutting board.

[0341] In this way, when the information processing device 10 misrecognizes an ingredient in a cooking procedure recording application for cooking, it displays gesture pictograms corresponding to other ingredient options along with the corresponding options based on the user's correction gesture, prompting the user to make a correction. Then, by having the user select the correct ingredient, the correct cooking procedure can be displayed. Furthermore, since selection can be made using gestures made in the air, the user can correct the misrecognition by the information processing device 10 immediately, without any manual intervention, even if, for example, the user's hands are dirty while cooking.

[0342] <<9. Modifications>> Note that, among the processes described in the above-described embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the process procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0343] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0344] The above-described embodiments of the present disclosure can be combined as appropriate within the scope of the present disclosure without causing any contradiction in the processing content. The order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate.

[0345] <<10. Hardware Configuration>> The information processing device 10 according to the embodiment of the present disclosure described above is realized by, for example, a computer 1000 configured as shown in Fig. 59. Fig. 59 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device 10. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0346] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0347] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0348] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records the programs according to this embodiment.

[0349] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0350] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0351] For example, when the computer 1000 functions as the information processing device 10, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the functions of the control unit 12. The secondary storage device 1400 stores the program according to the present disclosure and data in the storage unit 11. The CPU 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0352] <<11. Conclusion>> As described above, according to one embodiment of the present disclosure, the information processing device 10 is an information processing device configured to be able to present content and interactions related to the progress of the content to a presentation target, and includes a control unit 12. The control unit 12 acquires image information having at least the presentation target as a capture target range. The control unit 12 recognizes the state of an object, including a user's action, in the recognition target range by performing a recognition process on a recognition target range in the image information. The control unit 12 determines whether to present the interaction associated with the recognition result based on the recognition result, the user's action, and rule information related to the content. The control unit 12 presents the interaction to the presentation target in accordance with the determination result. This prevents the progress of the content presented to the presentation target from being hindered by erroneous recognition of a real object related to the content.

[0353] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0354] Furthermore, the effects of each embodiment described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0355] The present technology can also be configured as follows. (1) An information processing device configured to present content and interactions related to progress of the content to a presentation target, the information processing device comprising: a control unit that acquires image information having at least the presentation target as a capture target range; recognizes a state of an object in the recognition target range, including a user's action, by performing a recognition process on a recognition target range in the image information; determines whether to present the interaction associated with the recognition result based on the recognition result, the user's action, and rule information related to the content; and presents the interaction to the presentation target in accordance with the determination result. (2) The information processing device described in (1), wherein the control unit, when recognizing that the user does not follow the presented interaction, determines an interpretation result corresponding to the recognition result based on the recognition result and the user's action. (3) The information processing device described in (2), wherein the control unit, when the accuracy of the recognition result is less than a predetermined threshold, determines that the user's action is correct and changes the interpretation result to content corresponding to the user's action. (4) The information processing device according to (2), wherein the control unit, when the accuracy of the recognition result is less than a predetermined threshold or when it recognizes that the user has requested a correction to the presented interaction, presents the interaction to the user to make the correction, determines that the user's action based on the interaction is correct, and changes the interpretation result to content corresponding to the user's action. (5) The information processing device according to (3) or (4), wherein the control unit, when changing the interpretation result, presents the interaction associated with the interpretation result. (6) The information processing device according to (3), (4), or (5), wherein the control unit, when determining that the user's action is correct, locks the recognition processing for a predetermined period of time. (7) The information processing device according to (6), wherein the control unit adds a predetermined amount to the predetermined period each time the control unit determines that the user's action is correct.(8) The information processing device according to (7), wherein the control unit sets the predetermined period in units of time or frames of the image information. (9) The information processing device according to (1), wherein the control unit determines the recognition result to be correct when it recognizes that the user has followed the presented interaction. (10) The information processing device according to (1), wherein the control unit presents the interaction associated with the recognition result when the accuracy of the recognition result is equal to or greater than a predetermined threshold. (11) The information processing device according to (10), wherein, when recognizing a state of the object that does not conform to the rule information, the control unit subtracts a predetermined deduction value according to the state of the object from the accuracy of the recognition result, and determines whether to present the interaction associated with the recognition result based on whether the accuracy of the recognition result after the subtraction is equal to or greater than a predetermined threshold. (12) The information processing device according to (11), wherein the control unit presents the interaction associated with the recognition result when the accuracy of the recognition result after the subtraction is equal to or greater than a predetermined threshold. (13) The information processing device according to (11) or (12), wherein the control unit deletes an interpretation result corresponding to the recognition result when the accuracy of the recognition result after the subtraction is less than a predetermined threshold. (14) The information processing device according to (11), (12), or (13), wherein the control unit, when recognizing a state of the object that does not conform to the rule information, presents the interaction to the user to make a correction, and subtracts the accuracy of the recognition result each time the interaction is ignored by the user. (15) The information processing device according to (14), wherein the control unit changes the interpretation result corresponding to the recognition result to content that conforms to the rule information when the accuracy of the recognition result after the subtraction is less than a predetermined threshold. (16) The information processing device according to (1), wherein the control unit normally performs a first recognition process as the recognition process, and, when recognizing that the user has requested a correction for the presented interaction, performs a second recognition process that is a recognition process with higher accuracy than the first recognition process.(17) The information processing device according to (16), wherein the control unit locks the recognition process again for a predetermined period of time according to the recognition result of the first recognition process, and after executing the second recognition process, locks the recognition process again for a period of time longer than the predetermined period. (18) The information processing device according to any one of (1) to (17), wherein the content is game content of a board game. (19) An information processing method executed by an information processing device configured to be able to present content and interactions related to progress of the content to a presentation target, the information processing method including: acquiring image information having at least the presentation target as a shooting target range; recognizing a state of an object including a user's action in the recognition target range by executing recognition processing on a recognition target range in the image information; determining whether to present the interaction associated with the recognition result based on the recognized recognition result, the user's action, and rule information related to the content; and presenting the interaction to the presentation target according to the determination result. (20) A computer-readable recording medium having a program recorded thereon, the program causing a computer configured to be able to present content and interactions related to the progress of the content to a presentation target to perform the following processes: acquire image information with at least the presentation target as a shooting target range; recognize the state of an object including a user's actions in the recognition target range by performing recognition processing on the recognition target range in the image information; determine whether or not to present the interaction associated with the recognition result based on the recognition result, the user's actions, and rule information related to the content; and present the interaction to the presentation target according to the determination result.

[0356] REFERENCE SIGNS LIST 10 Information processing device 11 Storage unit 11a Recognition model 11b Scoring definition information 11c Locking seconds definition information 11d Recognition interpretation information DB 11e Content DB 12 Control unit 12a Image acquisition unit 12b Recognition processing unit 12c Determination processing unit 12d Presentation processing unit

Claims

1. An information processing device capable of presenting content and interactions related to the progress of the content to a presentation target, comprising: a control unit that acquires image information having at least the presentation target as a shooting target range; recognizes the state of an object including a user's actions in the recognition target range by executing recognition processing on the recognition target range in the image information; determines whether or not to present the interaction associated with the recognition result based on the recognition result, the user's actions, and rule information related to the content; and presents the interaction to the presentation target in accordance with the judgment result.

2. The information processing device according to claim 1, wherein the control unit, when recognizing that the user does not comply with the presented interaction, determines an interpretation result corresponding to the recognition result based on the recognition result and the user's actions.

3. The information processing device according to claim 2, wherein the control unit determines that the user's action is correct when the accuracy of the recognition result is less than a predetermined threshold, and changes the interpretation result to content corresponding to the user's action.

4. The information processing device of claim 2, wherein the control unit, when the accuracy of the recognition result is less than a predetermined threshold or when it recognizes that the user has requested a correction to the presented interaction, presents the interaction to the user to allow the user to make a correction, determines that the user's action based on the interaction is correct, and changes the interpretation result to content corresponding to the user's action.

5. The information processing device according to claim 3, wherein the control unit, when changing the interpretation result, presents the interaction associated with the interpretation result.

6. The information processing device according to claim 3, wherein the control unit, when determining that the user's action is correct, locks the recognition process for a predetermined period of time.

7. The information processing device according to claim 6, wherein the control unit adds a predetermined amount to the predetermined period each time the user's action is determined to be positive.

8. The information processing device according to claim 7, wherein the control unit sets the predetermined period in units of time or in units of frames of the image information.

9. The information processing device according to claim 1, wherein the control unit determines that the recognition result is correct when it recognizes that the user has followed the presented interaction.

10. The information processing device according to claim 1, wherein the control unit presents the interaction associated with the recognition result when the accuracy of the recognition result is equal to or greater than a predetermined threshold.

11. The information processing device described in claim 10, wherein the control unit, when recognizing a state of the object that does not conform to the rule information, subtracts a predetermined deduction value corresponding to the state of the object from the accuracy of the recognition result, and determines whether to present the interaction associated with the recognition result based on whether the accuracy of the recognition result after the subtraction is equal to or greater than a predetermined threshold.

12. The information processing device according to claim 11, wherein the control unit presents the interaction associated with the recognition result when the accuracy of the recognition result after the subtraction is equal to or greater than a predetermined threshold.

13. The information processing device according to claim 11, wherein the control unit deletes the interpretation result corresponding to the recognition result when the accuracy of the recognition result after the subtraction is less than a predetermined threshold.

14. The information processing device of claim 11, wherein the control unit, when recognizing a state of the object that does not conform to the rule information, presents the interaction to the user to allow the user to make a correction, and subtracts the accuracy of the recognition result each time the interaction is ignored by the user.

15. The information processing device according to claim 14, wherein the control unit, when the accuracy of the recognition result after the subtraction is less than a predetermined threshold, changes an interpretation result corresponding to the recognition result to content in accordance with the rule information.

16. The information processing device according to claim 1, wherein the control unit normally executes a first recognition process as the recognition process, and when it recognizes that the user has requested a correction to the presented interaction, executes a second recognition process which is a recognition process with higher accuracy than the first recognition process.

17. The information processing device according to claim 16, wherein the control unit locks the re-recognition process for a predetermined period of time depending on the recognition result of the first recognition process, and locks the re-recognition process for a period longer than the predetermined period of time after executing the second recognition process.

18. The information processing device according to claim 1, wherein the content is game content for a board game.

19. An information processing method executed by an information processing device capable of presenting content and interactions related to the progress of the content to a presentation target, the information processing method including: acquiring image information having at least the presentation target as a shooting target range; recognizing a state of an object including a user's actions in the recognition target range by performing recognition processing on a recognition target range in the image information; determining whether or not to present the interaction associated with the recognition result based on the recognized recognition result, the user's actions, and rule information related to the content; and presenting the interaction to the presentation target in accordance with the judgment result.

20. A computer-readable recording medium having a program recorded thereon, the program causing a computer configured to present content and interactions related to the progress of the content to a presentation target to execute the following processes: acquire image information with at least the presentation target as a shooting target range, recognize the state of an object including the user's actions in the recognition target range by executing recognition processing on the recognition target range in the image information, determine whether or not to present the interaction associated with the recognition result based on the recognition result, the user's actions, and rule information related to the content, and present the interaction to the presentation target depending on the result of the determination.

Citation Information

Patent Citations

  • Growing game program, growing game control method, and growing game apparatus

    JP2010162176A

  • Method for fading out image of physical object

    JP2017148530A

  • Program, information processing device and information processing method

    JP2020162840A

  • Virtual space display device and program

    JP2021176460A

  • Display system and server

    JP2022097830A