Information processing device and information processing method

Abstract

An information processing device 18 is an information terminal that can be attached to and detached from a controller 16 of a game console 12 that executes an application. The information processing device 18 comprises an attachment member for attaching the information processing device 18 to the controller 16. The information processing device 18 detects an operation or a state of a target object 100 representing an object that is different from the controller 16 and is attached to the information processing device 18 by means of the attachment member. The information processing device 18 outputs information based on the detection result.

Description

Information Processing Apparatus and Information Processing Method , ,

[0004] ,

[0008] ,

[0007] , ,

[0006] , ,

[0005] ,

[0001] The present disclosure relates to data processing technology, and particularly to an information processing apparatus and an information processing method.

[0002] A wide variety of game applications are provided.

[0003] There is a demand to provide users with a novel gaming experience. One object of the present disclosure is to provide a technology that supports the provision of a novel gaming experience.

[0004] In order to solve the above problems, an information processing apparatus according to an aspect of the present invention is an information processing apparatus detachable from a controller of a game machine that executes an application, and includes an attachment member for attaching the present information processing apparatus to the controller, a sensor unit that detects the operation or state of an object different from the controller, which is attached to the present information processing apparatus by the attachment member, and an output unit that outputs information based on the detection result by the sensor unit.

[0005] Another aspect of the present invention is an information processing method. This method is an information processing apparatus that is detachable from a controller of a game machine that executes an application and includes an attachment member for attaching the present information processing apparatus to the controller, and the information processing apparatus detects the operation or state of an object different from the controller, which is attached to the present information processing apparatus by the attachment member, and outputs information based on the result of the detection.

[0006] In addition, any combination of the above components, and those obtained by converting the expression of the present disclosure among a system, a computer program, a recording medium storing the computer program, etc. are also effective as aspects of the present disclosure.

[0007] According to the present disclosure, it is possible to support the provision of a novel gaming experience.

[0008] This is a diagram showing the configuration of the game system according to the first embodiment. This is a diagram showing the schematic appearance of the information processing device from the front and the right side. This is a diagram showing the schematic appearance of the controller and information processing device from the front. This is a block diagram showing the functional blocks of the information processing device according to the first embodiment. This is a block diagram showing the functional blocks of the information processing device according to the second embodiment. This is a diagram showing the schematic appearance of the controller and information processing device. This is a diagram showing an example of function mapping by the user. This is a block diagram showing the functional blocks of the information processing device according to the third embodiment. This is a diagram schematically showing the inside of a clip. This is a diagram showing an example of measuring the temperature of food placed in a ladle. This is a diagram showing an example of hardness measurement. This is a block diagram showing the functional blocks of the information processing device according to the fourth embodiment. This is a block diagram showing the functional blocks of the game machine according to the fifth embodiment. This is a block diagram showing the functional blocks of the controller according to the sixth embodiment.

[0009] <First Embodiment> Figure 1 shows the configuration of a game system 10 according to the first embodiment. The game system 10 is an information processing system comprising a game machine 12, a display device 14, a controller 16, and an information processing device 18.

[0010] The functions of the components in the game console 12, controller 16, and information processing device 18 may be realized in circuits or processing circuits, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, which are configured or programmed to realize the functions described herein. A processor is considered to be a circuit or processing circuit that includes transistors and other circuits. A processor may also be a programmed processor that executes a program stored in memory.

[0011] In this specification, circuits, units, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0012] If the hardware is a processor that is considered to be a type of circuit, then the circuit, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0013] The game console 12 is an information processing device including a computer capable of running various application software such as game applications. The game console 12 in the first embodiment is a stationary game console.

[0014] The controller 16 is an operating device that receives user inputs to system software and application software (e.g., video games) executed by the game console 12. The user inputs operations by grasping the controller 16. The controller 16 is connected to the game console 12 by wire or wireless connection and sequentially transmits operation information indicating the operations input by the user to the game console 12.

[0015] The controller 16 is equipped with a speaker and can output sound based on audio signals transmitted from the game console 12 (game application, etc.). The controller 16 is also equipped with a vibrator and can vibrate the vibrator in a manner based on haptic signals transmitted from the game console 12 (game application, etc.).

[0016] The display device 14 has a display that outputs images and a speaker that outputs sound. The display device 14 may be a television or a computer display. The display device 14 is connected to the game console 12 by wire or wireless connection. When the game console 12 receives operation information provided by the controller 16, it reflects that operation information in the processing of the system software and application software, displays an image related to the processing result on the display device 14, and outputs sound related to the processing result from the speaker of the display device 14.

[0017] The information processing device 18 is a small information terminal that can be attached to and detached from the controller 16. Furthermore, the information processing device 18 can also be attached to an object different from the controller 16 (hereinafter also referred to as "object 100"). In the first embodiment, the information processing device 18 changes its function as an operation input means depending on the object 100 to which it is attached.

[0018] Figure 2 shows a schematic appearance of the information processing device 18 from the front and right side. As shown in the front view, the information processing device 18 includes a display unit 20 on which various content such as text and images are displayed. Also, as shown in the right side view, the information processing device 18 includes a mounting member 21 for attaching the information processing device 18 to the controller 16 or the object 100. The mounting member 21 includes a clip 22 and a hinge 24. The hinge 24 serves as the pivot point for opening and closing the clip 22. The clip 22 includes a protrusion 26 for gripping the object 100.

[0019] Figure 3 schematically shows the general appearance of the controller 16 and information processing device 18 from the front. The controller 16 includes multiple buttons (face buttons, directional keys, etc.), an analog stick, and a mounting part 17. The mounting part 17 is located in the center of the controller 16 and is a component to which the information processing device 18 can be attached. The display unit 20 in Figure 3 displays content 102, including a character that acts as the user's companion.

[0020] Figure 4 is a block diagram showing the functional blocks of the information processing device 18 of the first embodiment. Each block shown in the block diagrams of this specification can be realized in hardware terms by a computer processor (CPU, etc.), memory and other elements, electronic circuits, and mechanical devices, and in software terms by a computer program, etc., but here we are depicting functional blocks that are realized through the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by combinations of hardware and software.

[0021] The information processing device 18 includes a display unit 20, a sensor 30, a processing unit 32, a storage unit 34, and a communication unit 36. The display unit 20 may include a liquid crystal display or an organic EL display. The display unit 20 may also include a touch panel and may function as an operation input unit.

[0022] The sensor 30 measures a predetermined physical quantity relating to the operation or state of an object 100, which is different from the controller 16 and is attached to the information processing device 18 by a mounting member 21. The sensor 30 may include at least one of the following: an inertial measurement unit (hereinafter also called an "IMU"), a microphone, a camera (image sensor), a conductivity meter, a touch sensor, a potentiometer, and a capacitance sensor. The IMU may include a three-axis acceleration sensor and a three-axis angular velocity sensor.

[0023] The processing unit 32 performs various data processing operations. The storage unit 34 stores data that is referenced or updated by the processing unit 32. The communication unit 36 ​​communicates with external devices according to a predetermined communication protocol. When the information processing device 18 is removed from the controller 16, the communication unit 36 ​​communicates wirelessly with the controller 16. The processing unit 32 sends and receives data with the controller 16 via the communication unit 36, and further sends and receives data with the game machine 12 via the controller 16.

[0024] The storage unit 34 includes a conversion rule storage unit 40 that stores predetermined conversion rules. The conversion rules are information that indicates rules for converting (A) the operation or state of an object 100 detected by the sensor 30 to (B) operation information to be input to the game machine 12 (in other words, operation information for an application executed on the game machine 12), or in other words, information that indicates the correspondence between (A) and (B). In the first embodiment, the conversion rules define operation information that can be input from the controller 16 to the game machine 12 as the operation information of (B). That is, in the first embodiment, operation information that can be input from the controller 16 to the game machine 12 can be input alternatively using the object 100 to which the information processing device 18 is attached.

[0025] The processing unit 32 includes an operation detection unit 42, an operation information generation unit 44, and an operation information output unit 46. A computer program implementing the functions of these multiple functional blocks may be installed in the storage of the information processing device 18. The processor of the information processing device 18 may perform the functions of these multiple functional blocks by reading this computer program into main memory and executing it.

[0026] The sensor 30 and the motion detection unit 42 function as a sensor unit that detects the operation or state of the object 100, which is attached to the information processing device 18 by the mounting member 21. Specifically, the motion detection unit 42 detects the operation or state of the object 100 based on the physical quantity measured by the sensor 30.

[0027] The operation information generation unit 44 and the operation information output unit 46 function as output units that output information based on the detection results from the sensor unit. Specifically, the operation information generation unit 44 generates operation information that corresponds to the detection results from the sensor unit (sensor 30 and motion detection unit 42) from among the operation information that can be input from the controller 16 to the game machine 12. The operation information output unit 46 transmits the operation information generated by the operation information generation unit 44 to the game machine 12.

[0028] The operation of the game system 10 of the first embodiment will be described below, divided into cases for each type of object 100 to which the information processing device 18 is attached.

[0029] Case 1: The object 100 to which the information processing device 18 is attached is a rod-shaped object that the user grasps and moves. The user attaches the information processing device 18 to the rod-shaped object and then grasps and moves the rod-shaped object (for example, by shaking it).

[0030] The sensor 30 of the information processing device 18 includes an IMU. The motion detection unit 42 of the information processing device 18 detects the motion of the object 100 (such as swinging a stick or tilting a stick) based on the measurement results from the IMU. The conversion rule associates the motion of the object 100 with operation information for the game machine 12. The operation information for the game machine 12 is, for example, information indicating the pressing of a specific face button on the controller 16, the pressing of a trigger button on the controller 16, or the tilting (or leaning) of the analog stick on the controller 16 in a specific direction.

[0031] The operation information generation unit 44 of the information processing device 18 generates operation information for the game machine 12 that corresponds to the operation mode of the object 100 detected by the operation detection unit 42, according to the conversion rules. The operation information output unit 46 of the information processing device 18 transmits the operation information generated by the operation information generation unit 44 to the controller 16. The controller 16 transfers the operation information to the game machine 12. The game machine 12 receives the operation information and proceeds with the game based on that operation information. According to Case 1, the movement of a rod-shaped object can be used as input to the game. For example, a rod-shaped object can be used as a drumstick in a music game.

[0032] As a variation, the sensor 30 of the information processing device 18 may include a microphone. The information processing device 18 may further include a voice recognition unit that recognizes the sound input to the microphone. The conversion rule may associate the movement pattern of the object 100 (in this case, a rod-shaped object) with the recognized sound and operation information for the game machine 12 (for example, an operation corresponding to a specific move in the game). The operation information generation unit 44 of the information processing device 18 may generate operation information for the game machine 12 that corresponds to the movement pattern of the object 100 (in this case, a rod-shaped object) with the recognized sound. According to this variation, a novel game experience can be provided in which the moves in the game change according to the words shouted by the user while moving the rod-shaped object.

[0033] In another modification, the sensor 30 of the information processing device 18 may include at least one of a camera, a conductivity meter, a color sensor (a sensor capable of recognizing color), and a magnetic sensor. The information processing device 18 may further include a material recognition unit that recognizes the material of the object 100 to which the information processing device 18 is attached, based on data output from at least one of the camera, conductivity meter, color sensor, and magnetic sensor. The conversion rule may associate the movement mode of the object 100 (in this case, a rod-shaped object) with the recognized material and operation information for the game machine 12 (for example, an operation corresponding to a specific technique in the game). The operation information generation unit 44 of the information processing device 18 may generate operation information for the game machine 12 that corresponds to the movement mode of the object 100 (in this case, a rod-shaped object) with the recognized material. According to this modification, a novel game experience can be provided in which the techniques (or their power) in the game change depending on the material of the rod-shaped object moved by the user.

[0034] Case 2: The object 100 to which the information processing device 18 is attached is the user's clothing or the user's body. The user attaches the information processing device 18 to their clothing or body and then moves their body (for example, jumping, marching in place, swinging their arms, etc.).

[0035] The sensor 30 of the information processing device 18 detects at least one of the following as movements of the user's clothing or the user's body (hereinafter also simply referred to as "movements of the object 100"): vertical movement (e.g., jumping), stepping, or arm rotation. The conversion rule may associate vertical movement as movement of the object 100 with a button operation that instructs a jump (e.g., pressing the X button). Alternatively, stepping as movement of the object 100 may be associated with an analog stick operation that instructs a forward movement (e.g., tilting forward). Furthermore, arm rotation as movement of the object 100 may be associated with a button operation that instructs an attack (e.g., pressing the R1 button).

[0036] The operation information generation unit 44 of the information processing device 18 generates operation information for the game machine 12 that corresponds to the operation mode of the object 100 detected by the operation detection unit 42, according to the conversion rules. The subsequent operations are the same as in Case 1. According to Case 2, the movement of the clothes worn by the user or the user's body movements can be used as input to the game (for example, operation input to instruct the movement of an in-game character).

[0037] Case 3: The object 100 to which the information processing device 18 is attached is a disc-shaped object. The user attaches the information processing device 18 to the disc-shaped object, then grasps the disc-shaped object and rotates it (this can also be called rotating it; the same applies hereinafter).

[0038] The sensor 30 of the information processing device 18 includes an IMU. The motion detection unit 42 of the information processing device 18 detects the rotation angle and angular velocity of the object 100 based on the measurement results from the IMU. The conversion rule associates the rotation angle and angular velocity of the object 100 with the left and right tilting operation of the analog stick of the controller 16 (for example, steering operation in a racing game).

[0039] The operation information generation unit 44 of the information processing device 18 generates operation information for the game machine 12 that corresponds to the rotation angle and angular velocity of the object 100 detected by the motion detection unit 42, according to the conversion rules. The operation information here is information indicating the left and right tilting of the analog stick of the controller 16. The subsequent operation is the same as in Case 1. According to Case 3, the movement of a disc-shaped object held by the user can be used as input to the game (for example, steering in a racing game).

[0040] Case 4: The object 100 to which the information processing device 18 is attached can be an object of various shapes (for example, a disc, a rod, or an object with a free curve). The user attaches the information processing device 18 to the object 100 and moves the information processing device 18 along the outer edge of the object 100.

[0041] The sensor 30 of the information processing device 18 includes at least one of an IMU, a camera, and a wireless device. The motion detection unit 42 of the information processing device 18 may detect changes in the position of the information processing device 18 using a position estimation technique based on at least one of the measurement results from the IMU, the image captured by the camera, and the wireless signal strength. The conversion rule is a rule that converts the shape drawn by the change in the position of the information processing device 18 into operation information for input to the game machine 12.

[0042] The operation information generation unit 44 of the information processing device 18 generates operation information that specifies the shape drawn by the change in the position of the information processing device 18 detected by the motion detection unit 42, according to the conversion rules. The subsequent operations are the same as in Case 1. According to Case 4, the shape of an object in real space can be input into the game world.

[0043] Case 5: The object 100 to which the information processing device 18 is attached is a hollow object. Hereafter, the object 100 will also be referred to as a "cylindrical object" as an example. The user attaches the information processing device 18 inside the cylindrical object and then rolls the cylindrical object.

[0044] The sensor 30 of the information processing device 18 includes an IMU. The motion detection unit 42 of the information processing device 18 detects the rotational motion (for example, the rotation angle and angular velocity) of the cylindrical object based on the measurement results by the IMU. The conversion rule associates the rotation angle and each velocity of the cylindrical object with an operation (for example, the operation of the left and right keys or the operation of the analog stick in the left and right directions) for rolling an object in the game.

[0045] The operation information generation unit 44 of the information processing device 18 generates operation information for rolling an object in the game corresponding to the rotation angle and angular velocity of the object 100 detected by the motion detection unit according to the conversion rule. The subsequent operations are the same as those in Case 1. According to Case 5, the rotational motion of the cylindrical object can be used as an input to the game (for example, an operation input for rolling an object in the game).

[0046] Case 6: The object 100 to which the information processing device 18 is attached is a consumable such as paper or aluminum foil. The user arbitrarily folds the consumable and sandwiches the folded consumable with the clip 22 of the information processing device 18.

[0047] The sensor 30 of the information processing device 18 includes a potentiometer that measures the opening degree of the clip 22. The motion detection unit 42 of the information processing device 18 detects the opening angle of the clip 22 based on the measurement results of the potentiometer. The conversion rule is a rule for converting the opening angle of the clip 22 into the thickness of the object 100 (folded), and is also a rule for converting the thickness of the object 100 after conversion into operation information for inputting to the game machine 12.

[0048] The operation information generation unit 44 of the information processing device 18 generates operation information for inputting the thickness of the object 100 to the game machine 12 corresponding to the opening angle of the clip 22 detected by the motion detection unit according to the conversion rule. The subsequent operations are the same as those in Case 1. According to Case 6, the user can change the object 100 serving as the input means into a shape he / she likes, and can provide the user with a once-in-a-lifetime game experience.

[0049] As a variation, if the object 100 is made of a conductive material such as aluminum foil, electrical contacts may be provided on the clip 22. This allows the object 100 to function as a touch sensor. The sensor 30 of the information processing device 18 may measure the voltage or current at the clip 22, and the operation information generation unit 44 of the information processing device 18 may generate operation information indicating that a predetermined touch operation has been performed (for example, operation information indicating that the touchpad on the controller 16 has been touched) when the change in voltage or current at the clip 22 satisfies predetermined conditions.

[0050] Case 7: The object 100 to which the information processing device 18 is attached is a base (in this case, a structure that can be stepped on). The user attaches the base to the clip 22 of the information processing device 18 and opens the clip 22 by stepping on the base.

[0051] The sensor 30 of the information processing device 18 includes a potentiometer attached to the clip 22 that measures the degree to which the clip 22 is opened (here referred to as the opening angle). The operation detection unit 42 of the information processing device 18 measures the degree to which the clip 22 is opened (opening angle) based on the measurement result from the potentiometer. The conversion rule associates the opening angle of the clip 22 with the operation of pressing the accelerator pedal of the vehicle in the game.

[0052] The operation information generation unit 44 of the information processing device 18 generates operation information indicating the operation of pressing the accelerator of the vehicle in the game, corresponding to the opening angle of the clip 22 detected by the operation detection unit 42, according to the conversion rules. The operation information generation unit 44 may set the operation information so that the force with which the accelerator of the vehicle in the game is pressed increases as the opening angle of the clip 22 increases. The subsequent operations are the same as in Case 1. According to Case 7, a novel game experience can be provided in which the vehicle in the game is controlled using an object 100 that is close to the user, with operations similar to those in the real world.

[0053] Case 8: The object 100 to which the information processing device 18 is attached is any object that moves actively or passively. The user attaches the information processing device 18 to the object 100. The user may also move the object 100 to which the information processing device 18 is attached.

[0054] The sensor 30 of the information processing device 18 includes at least one of a camera, an IMU, and a wireless device. The motion detection unit 42 of the information processing device 18 estimates the position of the object 100 using a position estimation technique based on at least one of the measurement results from the IMU, the image captured by the camera, and the wireless signal strength. The conversion rule is a rule that converts the position (or change thereof) of the object 100 into operation information for input to the game machine 12.

[0055] The operation information generation unit 44 of the information processing device 18 generates operation information that specifies the position (or change thereof) of the object 100 detected by the motion detection unit 42, according to the conversion rules. The subsequent operations are the same as in Case 1. According to Case 8, the movement of the object 100 in the real world can be reproduced in the game world.

[0056] According to the information processing device 18 of the first embodiment, a novel gaming experience can be provided by outputting information based on the operation or state of the object 100 attached by the clip 22. For example, operation information based on the operation or state of the object 100 to which the information processing device 18 is attached can be input to the game machine 12, thereby improving the enjoyment of the game.

[0057] The present disclosure has been described above based on the first embodiment. The first embodiment is illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of each component and each processing process, and that such modifications are also within the scope of the present disclosure.

[0058] Modifications relating to the first embodiment will now be described. The information processing device 18 may be capable of executing multiple types of operations from among the operations shown in Cases 1 to 8 of the first embodiment. The operation detection unit 42, operation information generation unit 44, and operation information output unit 46 of the information processing device 18 may switch functions according to the user's selected operation so as to execute the type of operation selected by the user from among the multiple types of operations.

[0059] <Second Embodiment> The second embodiment of this disclosure will be described focusing on the differences from the first embodiment. Components of the second embodiment that are the same as or equivalent to components of the first embodiment will be denoted by the same reference numerals as those of the first embodiment, and explanations that overlap with the description of the first embodiment will be omitted as appropriate.

[0060] The configuration of the game system 10 in the second embodiment is the same as the configuration of the game system 10 in the first embodiment shown in Figure 1. In the second embodiment, the object 100 attached to the information processing device 18 is a sheet-like object on which one or more objects to be operated by the user are drawn. In the second embodiment, the sheet-like object is made of paper, but as a modification, it may be made of something other than paper, such as plastic or film. In the second embodiment, a technique is proposed to make the paper attached to the information processing device 18 function as a controller.

[0061] Figure 5 is a block diagram showing the functional blocks of the information processing device 18 of the second embodiment. The information processing device 18 of the second embodiment differs from the information processing device 18 of the first embodiment in that it includes a mapping unit 47 and an operation target detection unit 48 instead of an operation detection unit 42.

[0062] The mapping unit 47 maps the objects to be operated drawn on the paper 110 to the actual operation elements to be input to the game machine 12 (for example, functional elements such as buttons on the controller 16), and stores the result of the mapping in the conversion rule storage unit 40.

[0063] The sensor 30 and the target detection unit 48 function as a sensor unit that detects the target that has been controlled by the user from among the targets drawn on the object 100. Specifically, the sensor 30 measures physical quantities related to the target that has been controlled by the user. Based on the physical quantities measured by the sensor 30, the target detection unit 48 detects the target that has been controlled by the user from among the multiple targets drawn on the paper 110.

[0064] The operation information generation unit 44 and the operation information output unit 46 function as output units that transmit operation information based on the target of operation input by the user to the game console 12. Specifically, the operation information generation unit 44 generates operation information based on the target of operation input by the user, which is detected by the target of operation detection unit 48. The operation information output unit 46 inputs an operation to the game application running on the game console 12 by transmitting the operation information generated by the operation information generation unit 44 to the game console 12.

[0065] The operation of the game system 10 of the second embodiment will now be described. Figure 6 shows an example of an information processing device 18 and paper 110. The user draws multiple controllable objects on the paper 110. The multiple controllable objects include a button 112a (hereinafter also referred to as "button 1"), a button 112b (hereinafter also referred to as "button 2"), a button 112c (hereinafter also referred to as "button 3"), and a slider 114. The slider 114 is a controllable object in which the user moves a knob 116 left and right and inputs a value corresponding to the position of the knob 116. The controllable objects can also be said to be functional elements of the controller.

[0066] The sensor 30 of the information processing device 18 may include a Time of Flight (ToF) sensor array or a depth camera as a sensor for sensing the position (X, D) of the user's finger on the paper 110 through the sensor surface 31. The depth camera may be a camera that acquires data in the lateral direction (X direction), height direction (Y direction), and depth direction (D direction) of the object, or a camera that acquires data in the lateral direction (X direction) and depth direction (D direction) of the object. The sensor 30 may also include an IMU or an impact sensor as a sensor for detecting when a finger touches the paper 110.

[0067] Method 1 or Method 2 below may be used to map the object to be operated drawn on paper 110 (for example, drawn button 1, button 2, etc.) to the actual operation element (for example, button 1, button 2, etc. provided on controller 16).

[0068] Method 1: The user may take a photograph of the paper 110 with a camera such as a smartphone. The mapping unit 47 of the information processing device 18 may analyze the captured image and recognize the shape and pattern (including marks such as "1" and "2") of the object to be operated on the paper 110, thereby mapping the position or area of ​​the object to be operated to the actual operation function. The mapping unit 47 may store the mapping result as a conversion rule in the conversion rule storage unit 40.

[0069] Method 2: The mapping unit 47 of the information processing device 18 may display the screen of a utility application for function mapping on the display unit 20. The user may input definitions of multiple areas that divide the paper 110 (for example, specifying ranges using the horizontal direction X and the depth direction D) on the utility application screen. The user may also specify the actual operation functions to be assigned to each area on the utility application screen.

[0070] Figure 7 shows an example of user-defined function mapping. In the example in Figure 7, button 1 is assigned to area 1 in the upper left, button 2 is assigned to area 2 in the upper right, and a slider is assigned to area 3 at the bottom. The mapping unit 47 of the information processing device 18 may store the mapping information entered on the utility application screen as a conversion rule in the conversion rule storage unit 40.

[0071] The mapping unit 47 may be implemented in the game machine 12. In that case, the game machine 12 may transmit the mapping information (conversion rules) generated by the mapping unit 47 to the information processing device 18, and the information processing device 18 may store the mapping information (conversion rules) transmitted from the game machine 12 in the conversion rule storage unit 40.

[0072] The user clips the paper 110, on which the object to be manipulated is drawn, with the clip 22 of the information processing device 18. The user touches the desired object to be manipulated on the paper 110 with their finger. If the object to be manipulated is a slider 114, the user may also trace the knob 116 to slide it to the desired position.

[0073] The object detection unit 48 of the information processing device 18 detects that the user's finger has come into contact with the paper 110 based on the measurement results from the sensor 30. For example, if the sensor 30 includes a depth camera capable of detecting positions in the X, Y, and D directions, the object detection unit 48 may detect that the user's finger has come into contact with the paper 110 based on the value in the Y direction (for example, whether the height is 0 or not). On the other hand, if the sensor 30 includes a depth camera or ToF sensor array capable of detecting positions in the X and D directions, the object detection unit 48 may detect that the user's finger has come into contact with the paper 110 based on the measurement results from a separately provided IMU or impact sensor.

[0074] When the system detects that the user's finger has come into contact with the paper 110, the operation target detection unit 48 detects the position of contact based on the measurement results from the sensor 30. For example, if the sensor 30 includes a ToF sensor array, the operation target detection unit 48 may identify the sensor number that detected the distance as the lateral contact position (X) and the measurement value from the ToF sensor array as the depth contact position (Z). On the other hand, if the sensor 30 includes a depth camera, the operation target detection unit 48 may identify the lateral position (X) and depth position (Z) measured by the depth camera as the contact position. The operation target detection unit 48 identifies the operation target object corresponding to the position of contact with the user's finger as the operation target object to which the user's operation was input, according to the conversion rules.

[0075] The operation information generation unit 44 of the information processing device 18 identifies the actual operation element corresponding to the operation target object to which the user's operation has been input, as identified by the operation target detection unit 48, according to the conversion rules, and generates operation information indicating that operation element. For example, if the user's finger touches button 1 drawn on paper 110, the operation information generation unit 44 generates operation information indicating that button 1 on the controller 16 has been pressed. If the operation target object to which the user's operation has been input is a slider 114, the operation information generation unit 44 may also generate operation information indicating that the knob 116 is set to the finger release position.

[0076] The operation information output unit 46 of the information processing device 18 transmits the operation information generated by the operation information generation unit 44 to the controller 16. The controller 16 transfers the operation information to the game machine 12. The game machine 12 receives the operation information and executes the game application based on that information.

[0077] According to the information processing device 18 of the second embodiment, operation information based on the user's operation on the sheet-like object to which the information processing device 18 is attached can be input to the game machine 12, providing the user with a novel gaming experience. In addition, the user can place the object to be operated at a desired position on the paper 110, increasing the degree of freedom of operation.

[0078] The present disclosure has been described above based on the second embodiment. The second embodiment is illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of each component and each processing process, and that such modifications are also within the scope of the present disclosure.

[0079] Modifications relating to the first and second embodiments will now be described. The processing unit 32 of the information processing device 18 may further include a display control unit that controls the display content on the display unit 20. The display control unit may display on the display unit 20 information relating to operation information generated by the operation information generation unit 44, or in other words, information relating to operation information transmitted from the information processing device 18 to the game machine 12.

[0080] Information regarding operation may include text or images indicating that an operation was input via object 100. Furthermore, information regarding operation may also include text or images related to the operation indicated by the operation information (e.g., pressing a button). This makes it easy for the user to understand that an operation was input via object 100 and the operation being input to the game console 12.

[0081] Further modifications relating to the first and second embodiments will be described. The information processing device 18 may further include a sensor information storage unit that stores information based on the detection results of the sensor 30 (hereinafter also referred to as "sensor information"). The operation information output unit 46 may output to the outside sensor information based on prior detection results of the sensor 30, which has been stored in the sensor information storage unit in advance, when predetermined conditions are met.

[0082] For example, the operation information output unit 46 may transmit sensor information stored in the sensor information storage unit to the game machine 12 when sensor information is requested from the game machine 12 or when a predetermined user operation is input to the display unit 20. Alternatively, the operation information output unit 46 may display the sensor information stored in the sensor information storage unit on the display unit 20 when predetermined information is detected by the sensor 30 or when a predetermined user operation is input to the display unit 20. This modified version can support the provision of an even wider variety of game experiences.

[0083] <Third Embodiment> The third embodiment of this disclosure will be described focusing on the differences from the first embodiment. Components of the third embodiment that are the same as or equivalent to components of the first embodiment will be denoted by the same reference numerals as those of the first embodiment, and explanations that overlap with the description of the first embodiment will be omitted as appropriate.

[0084] The configuration of the game system 10 in the third embodiment is the same as the configuration of the game system 10 in the first embodiment shown in Figure 1. In the third embodiment, the object 100 attached to the information processing device 18 includes the final completed dish (food), the dish in the process of being cooked, and the ingredients and cooking utensils used by the user to cook (hereinafter, these are collectively referred to as "food, etc."). In the third embodiment, we propose a technology to realize a real cooking game linked to cooking in the real world.

[0085] Figure 8 is a block diagram showing the functional blocks of the information processing device 18 of the third embodiment. The sensor 30 may include at least one of the following: a temperature sensor, a distance sensor (e.g., a ToF sensor), an encoder, an acceleration sensor, an odor sensor (a sensor that detects the intensity of an odor), or a gas sensor (a sensor that detects the type and concentration of a gas). The game storage unit 50 stores data for cooking games.

[0086] The processing unit 32 includes a game execution unit 52, a state detection unit 54, an output information generation unit 56, and a display control unit 58. The game execution unit 52 reads data stored in the game memory unit 50 and executes the cooking game. The display control unit 58 controls the display content of the display unit 20. The display control unit 58 displays the cooking game content (text and images) generated by the game execution unit 52 on the display unit 20.

[0087] The sensor 30 and the state detection unit 54 function as a sensor unit that detects the state of food or other items to which the information processing device 18 is attached. Specifically, the sensor 30 measures physical quantities related to the state of food or other items. The state detection unit 54 detects the state of food or other items based on the physical quantities measured by the sensor 30.

[0088] The output information generation unit 56 and the display control unit 58 function as output units that display information based on the state of food, etc., on the display unit 20. The output information generation unit 56 generates output information based on the state of food, etc., as content for the cooking game. The display control unit 58 displays the output information generated by the output information generation unit 56 on the display unit 20.

[0089] The operation of the game system 10 of the third embodiment will now be described. Figure 9 schematically shows the inside of the clip 22 (i.e., the side that clamps the object 100). Multiple temperature sensors 60 and multiple distance sensors 62 are arranged in a line along the longitudinal direction of the clip 22. Multiple temperature sensors 60 are provided on multiple protrusions 26. Distance sensors 62a, 62b, and 62c are provided at locations other than the protrusions 26. Distance sensors 62a and 62c are provided at the ends of the clip 22, and distance sensor 62b is provided in the center of the clip 22. In addition, a rotation angle sensor (an encoder in the third embodiment) is provided on the hinge 24 of the clip 22.

[0090] The user starts a cooking game on the information processing device 18. The game execution unit 52 of the information processing device 18 executes the cooking game and displays the cooking game content on the display unit 20. The cooking game content includes instructions to the user regarding the temperature, hardness, thickness, and cooking speed of food, etc., when the user creates a predetermined dish in the real world. The user creates the dish according to the instructions of the cooking game. The sensor 30 and state detection unit 54 of the information processing device 18 measure the temperature, hardness, thickness, and cooking speed of the food, etc. Examples of temperature measurement, hardness measurement, thickness measurement, and cooking speed measurement are described below.

[0091] Temperature Measurement: The user places the food item to be measured using the clip 22 of the information processing device 18, following the instructions of the cooking game. The temperature sensor 60 outputs the measurement result of a physical quantity related to the temperature of the food item that is in contact with the protrusion 26 of the clip 22. The state detection unit 54 detects the temperature of the food item based on the measurement result of the temperature sensor 60.

[0092] Figure 10 shows an example of measuring the temperature of food in a ladle. By clipping the metal part of the ladle handle with the clip 22 of the information processing device 18, the approximate temperature of the food (e.g., soup or sauce) in the ladle can be measured. The state detection unit 54 may also maintain a correspondence between the temperature indicated by the measurement result of the temperature sensor 60 and the temperature of the tip of the ladle, which is derived in advance based on the material and thermal conductivity of the ladle. The state detection unit 54 may also identify the temperature associated with the temperature indicated by the measurement result of the temperature sensor 60 as the temperature of the food, etc., that is in contact with the ladle.

[0093] The output information generation unit 56 generates output information indicating the temperature of food, etc. The display control unit 58 displays the output information indicating the temperature of food, etc. on the display unit 20. The game execution unit 52 determines whether the measured temperature of food, etc. matches the temperature instructed to the user, and displays information related to the determination result on the display unit 20. For example, if the two match, information indicating mission completion may be displayed on the display unit 20. Note that matching may include both the two being the same and the difference between the two being below a predetermined threshold. The threshold in this disclosure may be determined by the developer's knowledge or by experiments using the game system 10.

[0094] Hardness Measurement: The user places the food item to be measured using the clip 22 of the information processing device 18. Distance sensors 62a, 62b, and 62c each output measurement results of physical quantities related to the distance to the food item being measured. Figure 11 shows an example of hardness measurement. The state detection unit 54 detects the distance Δ1 from distance sensor 62a to the food item based on the measurement result of distance sensor 62a. The state detection unit 54 detects the distance Δ2 from distance sensor 62b to the food item based on the measurement result of distance sensor 62b. The state detection unit 54 detects the distance Δ3 from distance sensor 62c to the food item based on the measurement result of distance sensor 62c.

[0095] The state detection unit 54 determines that the food is soft if (1) the difference between Δ1 and Δ3 is less than a predetermined first threshold, (2) Δ2 is greater than Δ1 and the difference between Δ2 and Δ1 is greater than a predetermined second threshold (a threshold greater than the first threshold), and (3) Δ2 is greater than Δ3 and the difference between Δ2 and Δ3 is greater than the second threshold. On the other hand, the state detection unit 54 determines that the food is hard if (1) the difference between Δ1 and Δ2 is less than a first threshold, (2) the difference between Δ2 and Δ3 is less than a first threshold, and (3) the difference between Δ1 and Δ3 is less than a first threshold.

[0096] In the third embodiment, the state detection unit 54 maintains a predetermined correspondence between the difference between Δ1 and Δ2, the difference between Δ2 and Δ1, and the difference between Δ1 and Δ3, and the hardness of food, etc. The state detection unit 54 identifies the hardness associated with the detected difference between Δ1 and Δ2, the difference between Δ2 and Δ1, and the difference between Δ1 and Δ3 as the hardness of food, etc.

[0097] As a variation, if the information processing device 18 does not have a distance sensor, a contact detection sensor may be provided on the protrusion 26 of the clip 22, and the hardness of food, etc. may be measured based on the measurement result from the contact detection sensor and the measurement result from the encoder provided on the hinge 24. Specifically, the state detection unit 54 may maintain a predetermined correspondence between the amount of angle change of the clip 22 from the time contact of food, etc. is detected on the protrusion 26 of the clip 22 until the clip 22 stops moving, and the hardness of food, etc. The state detection unit 54 may detect the amount of angle change of the clip 22 from the time contact of food, etc. is detected by the contact detection sensor until the clip 22 stops moving, based on the output of the encoding, and may identify the hardness associated with the detected amount of angle change as the hardness of food, etc. that is clipped by the clip 22.

[0098] The output information generation unit 56 generates output information indicating the hardness of food, etc. The display control unit 58 displays the output information indicating the hardness of food, etc. on the display unit 20. The game execution unit 52 determines whether the measured hardness of food, etc. matches the hardness instructed to the user, and displays information related to the determination result on the display unit 20. For example, if both match, information indicating mission completion may be displayed on the display unit 20.

[0099] Thickness Measurement: The user places the food item to be measured into the clip 22 of the information processing device 18. An encoder provided on the hinge 24 outputs the measurement result of the angle at which the clip 22 is opened (in other words, the amount of angle change). The state detection unit 54 maintains a predetermined correspondence between the angle at which the clip 22 is opened and the thickness of the food item. The state detection unit 54 identifies the thickness associated with the angle at which the clip 22 is opened, as measured by the encoder, as the thickness of the food item.

[0100] The output information generation unit 56 generates output information indicating the thickness of food, etc. The display control unit 58 displays the output information indicating the thickness of food, etc. on the display unit 20. The game execution unit 52 determines whether the measured thickness of food, etc. matches the thickness instructed to the user, and displays information related to the determination result on the display unit 20. For example, if both match, information indicating mission completion may be displayed on the display unit 20.

[0101] Cooking speed measurement: A cooking utensil (e.g., a knife or whisk) is clipped to the information processing device 18 using the clip 22. The acceleration sensor of the information processing device 18 outputs the acceleration measurement result. The state detection unit 54 maintains a predetermined correspondence between the acceleration measurement result and the operating speed of the cooking utensil (e.g., the speed of julienning with a knife or the speed of stirring with a whisk). The state detection unit 54 identifies the operating speed corresponding to the acceleration measured by the acceleration sensor as the operating speed of the cooking utensil. The state detection unit 54 may also identify the result of integrating the acceleration measured by the acceleration sensor as the operating speed of the cooking utensil.

[0102] The output information generation unit 56 generates output information indicating the operating speed of the cooking appliance (for example, the speed of shredding or stirring). The display control unit 58 displays the output information indicating the operating speed of the cooking appliance on the display unit 20. The game execution unit 52 determines whether the measured operating speed of the cooking appliance matches the operating speed of the cooking appliance instructed to the user, and displays information related to the determination result on the display unit 20. For example, if both match, information indicating that the mission has been completed may be displayed on the display unit 20.

[0103] According to the information processing device 18 of the third embodiment, a novel gaming experience can be provided by displaying information based on the state of food or other items to which the information processing device 18 is attached. For example, a realistic cooking game linked to cooking in the real world can be realized.

[0104] The present disclosure has been described above based on the third embodiment. The third embodiment is illustrative, and it will be understood by those skilled in the art that various modifications are possible in combination with each component and each processing process, and that such modifications are also within the scope of the present disclosure.

[0105] <Fourth Embodiment> The fourth embodiment of this disclosure will be described focusing on the differences from the third embodiment. Components of the fourth embodiment that are the same as or equivalent to those of the third embodiment will be denoted by the same reference numerals as those of the third embodiment, and explanations that overlap with the explanation of the third embodiment will be omitted as appropriate.

[0106] The configuration of the game system 10 in the fourth embodiment is the same as the configuration of the game system 10 in the first embodiment shown in Figure 1. The information processing device 18 in the fourth embodiment has the functions of the information processing device 18 in the third embodiment, as well as the function of coordinating with a game application executed on the game machine 12.

[0107] Figure 12 is a block diagram showing the functional blocks of the information processing device 18 of the fourth embodiment. The information processing device 18 of the fourth embodiment includes, in addition to the functional blocks of the information processing device 18 of the third embodiment shown in Figure 8, a cooking history storage unit 64 and an information transmission unit 66. The information transmission unit 66 transmits information based on the state of food, etc., detected by the state detection unit 54 to the game machine 12. The cooking history storage unit 64 stores information based on the state of food, etc., detected by the state detection unit 54.

[0108] The operation of the game system 10 of the fourth embodiment will now be described. Similar to the third embodiment, while the user is cooking in the real world, the state detection unit 54 of the information processing device 18 detects the temperature, hardness, thickness, or cooking speed of the food attached by the clip 22. The state detection unit 54 stores the detected information on the temperature, hardness, thickness, or cooking speed of the food (hereinafter also referred to as "sensing information") in the cooking history storage unit 64. While the user is cooking in the real world, sensing information indicating the detection results by the state detection unit 54 is sequentially accumulated in the cooking history storage unit 64.

[0109] The information transmission unit 66 of the information processing device 18 transmits the sensing information stored in the cooking history storage unit 64 to the game machine 12 when predetermined notification conditions are met. The information transmission unit 66 may determine that the notification conditions are met when the information processing device 18 is attached to the controller 16. Alternatively, the information transmission unit 66 may determine that the notification conditions are met when a predetermined amount of time has elapsed since the last transmission of sensing information. Furthermore, a game application running on the game machine 12 may transmit information requesting the provision of sensing information to the information processing device 18. The information transmission unit 66 may determine that the notification conditions are met when it receives information requesting the provision of sensing information transmitted from the game machine 12.

[0110] The game application running on the game console 12 may receive sensing information transmitted from the information processing device 18 and reflect that sensing information in the progress of the game, or it may generate game content that reflects that sensing information. For example, if the sensing information transmitted from the information processing device 18 (e.g., temperature, hardness, thickness, cooking speed of food, etc.) satisfies the conditions for clearing an in-game mission, the game console 12 may proceed with the game as if the in-game mission has been cleared.

[0111] According to the information processing device 18 of the fourth embodiment, information based on the state of the food to which the information processing device 18 is attached becomes available at a later time than when the state is detected, enabling the realization of a highly engaging game. Furthermore, according to the information processing device 18, information based on the state of the food to which the information processing device 18 is attached can be input to the game machine, providing a novel gaming experience. For example, sensing information accumulated in the cooking history storage unit 64 during cooking is transmitted to the game machine 12 when the user completes the cooking and attaches the information processing device 18 to the controller 16, allowing the progress of the game executed on the game machine 12 to reflect the state of the cooking in the real world.

[0112] The present disclosure has been described above based on the fourth embodiment. The fourth embodiment is illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of each component and each processing process, and that such modifications are also within the scope of the present disclosure.

[0113] Modifications relating to the third and fourth embodiments will now be described. The game console 12 may transmit data for a cooking game, which is a mini-game within a game application executed by the game console 12, to the information processing device 18. The game memory unit 50 of the information processing device 18 may have the cooking game data transmitted from the game console 12 installed. The game application executed by the game console 12 may be implemented to receive the mission completion status of the cooking game, which is based on the state of real-world food, etc., transmitted from the information processing device 18, and to reflect the mission completion status of the cooking game in the progress of the game.

[0114] <Fifth Embodiment> The fifth embodiment of this disclosure will be described focusing on the differences from the first embodiment. Components of the fifth embodiment that are the same as or equivalent to those of the first embodiment will be denoted by the same reference numerals as those of the first embodiment, and explanations that overlap with the description of the first embodiment will be omitted as appropriate.

[0115] The configuration of the game system 10 in the fifth embodiment is the same as the configuration of the game system 10 in the first embodiment shown in Figure 1. In the game system 10 of the fifth embodiment, the game machine 12 determines the input to the application (video game, etc.) based on both the input from the controller 16 and the input from the information processing device 18.

[0116] Figure 13 is a block diagram showing the functional blocks of the game machine 12 according to the fifth embodiment. The game machine 12 comprises a processing unit 70, a storage unit 72, and a communication unit 74. The processing unit 70 performs various data processing. The storage unit 72 stores data that is referenced or updated by the processing unit 70. The communication unit 74 communicates with external devices according to a predetermined communication protocol. The processing unit 70 sends and receives data to and from the display device 14, controller 16, and information processing device 18 via the communication unit 74.

[0117] The storage unit 72 includes an application storage unit 76. The application storage unit 76 stores various application software (for example, video game applications) that are executed by the processing unit 70.

[0118] The processing unit 70 includes an application execution unit 80, a display control unit 82, an operation information receiving unit 84, and an operation determination unit 86. A computer program implementing the functions of these multiple functional blocks may be installed in the storage of the game console 12. The processor of the game console 12 may perform the functions of these multiple functional blocks by reading this computer program into main memory and executing it.

[0119] The application execution unit 80 executes the application program stored in the application storage unit 76 based on user operations. The application execution unit 80 generates an image (for example, an image of a video game) that shows the result of executing the application program. The display control unit 82 causes the display device 14 to display the image showing the result of executing the application program generated by the application execution unit 80.

[0120] The operation information receiving unit 84 receives first operation information, which is operation information generated and output by the controller 16, and second operation information, which is operation information generated and output by the information processing device 18. The first operation information includes information indicating the state of the controller 16 or an operation input to the controller 16. The state of the controller 16 may be detected by a sensor mounted on the controller 16 (e.g., an IMU). The second operation information includes information indicating the state of the information processing device 18 or an operation input to the information processing device 18. The state of the information processing device 18 may be detected by a sensor mounted on the information processing device 18 (e.g., an IMU). The second operation information may include operation information, output information, and sensing information output by the information processing device 18 in the first to fourth embodiments.

[0121] In a first embodiment, the information processing device 18 may be wirelessly connected to the game machine 12 and communicate directly with the game machine 12. In this case, the controller 16 may transmit first operation information to the game machine 12, and in parallel therewith, the information processing device 18 may transmit second operation information to the game machine 12 without going through the controller 16. In a second embodiment, the information processing device 18 may communicate indirectly with the game machine 12 via the controller 16. In this case, the information processing device 18 may transmit second operation information to the controller 16. The controller 16 may transmit first operation information to the game machine 12 and also forward the second operation information transmitted from the information processing device 18 to the game machine 12.

[0122] The operation determination unit 86 determines the information to be input to the running application (e.g., a video game application) based on the first operation information and the second operation information. The application execution unit 80 inputs the information determined by the operation determination unit 86, based on the first operation information and the second operation information (e.g., information regarding the operation and instructions of the application), to the running application and generates a new execution result (a new image, etc.) of the application. According to the game system 10 of the fifth embodiment, a highly engaging way of playing can be realized by coordinating the controller 16 and the information processing device 18.

[0123] The following describes examples (Examples 1-4) of determining input information to an application based on first operation information and second operation information. Examples 1-4 can be combined as appropriate. In each example, the controller 16 may be operated by a first user, and the information processing device 18 may be operated by a second user different from the first user.

[0124] Example 1: The operation decision unit 86 compares the input time of the first operation information with the input time of the second operation information, and if the difference between the two is less than or equal to a predetermined threshold (e.g., n seconds), it may decide to input information based on at least one of the first operation information and the second operation information into the application (in other words, it may permit it). The operation decision unit 86 may decide to input both the operation indicated by the first operation information and the operation indicated by the second operation information into the application, or it may decide to input only one of the operations into the application. If the difference between the input time of the first operation information and the input time of the second operation information exceeds the threshold, the operation decision unit 86 may restrict or prohibit the input of information based on at least one of the first operation information and the second operation information into the application.

[0125] The input times to be compared may be the time when the state is detected or the time when an operation is input for both the controller 16 and the information processing device 18, and the input time may be included in both the first operation information and the second operation information. Alternatively, the input time may be the time when both the first operation information and the second operation information are input to the game machine 12, or in other words, the time when the game machine 12 receives both the first operation information and the second operation information.

[0126] Example 2: The operation information receiving unit 84 may further acquire information detected by sensors mounted on the controller 16 and / or the information processing device 18 (hereinafter also referred to as "sensor information"). The sensor information may be included in either or both of the first operation information and the second operation information. The operation determination unit 86, as a detection unit, may detect the distance between the controller 16 and the information processing device 18 based on the sensor information acquired by the operation information receiving unit 84. Known techniques (for example, position measurement using wireless or distance measurement using a ToF camera) may be used to detect the distance.

[0127] The operation determination unit 86 may change the information to be input to the application based on the first operation information and the second operation information, depending on the distance between the controller 16 and the information processing device 18. For example, if the distance between the controller 16 and the information processing device 18 is less than a predetermined threshold (i.e., close), the operation determination unit 86 may determine "press the circle button" as the input information based on the first operation information and the second operation information. On the other hand, if the distance between the controller 16 and the information processing device 18 is greater than or equal to the threshold (i.e., far), the operation determination unit 86 may determine "press the X button" as the input information based on the first operation information and the second operation information.

[0128] In another embodiment, the operation determination unit 86 may convert the distance between the controller 16 and the information processing device 18 into the time the analog stick of the controller 16 is tilted. For example, if the distance between the controller 16 and the information processing device 18 is less than a predetermined threshold, the operation determination unit 86 may determine a short-duration tilt operation of the analog stick as input information. If the distance between the controller 16 and the information processing device 18 is greater than or equal to the threshold, the operation determination unit 86 may determine a long-duration tilt operation of the analog stick as input information.

[0129] Example 3: The operation determination unit 86 may determine the information to be input to the application based on the first operation information and the second operation information when the state or operation of the controller 16 indicated by the first operation information matches the state or operation of the information processing device 18 indicated by the second operation information. For example, the operation determination unit 86 may determine the rotation angle indicated by the first operation information or the second operation information as input information when the rotation angle of the controller 16 indicated by the first operation information matches the rotation angle of the information processing device 18 indicated by the second operation information. In other words, it may permit the input of the rotation angle indicated by the first operation information or the second operation information to the application. Note that "matching" includes the difference between the two being less than or equal to a predetermined threshold that can be considered a match.

[0130] The operation determination unit 86 may restrict or prohibit inputting information based on at least one of the first operation information and the second operation information into the application if the state or operation of the controller 16 indicated by the first operation information does not match the state or operation of the information processing device 18 indicated by the second operation information. For example, if the difference between the rotation angle of the controller 16 indicated by the first operation information and the rotation angle of the information processing device 18 indicated by the second operation information is greater than or equal to a predetermined threshold, the operation determination unit 86 may restrict or prohibit inputting the rotation angle indicated by the first operation information or the second operation information into the application.

[0131] Example 4: The operation determination unit 86 may decide to input information based on the first operation information into the application when the second operation information indicates a specific state or operation of the predetermined information processing device 18.

[0132] For example, a predetermined specific state or operation of the information processing device 18 (hereinafter also referred to as the "operation condition") may be the information processing device 18 moving up and down. When a second user holds the information processing device 18 and performs a squat, the second operation information indicates that the information processing device 18 is moving up and down. If the state or operation of the information processing device 18 indicated by the second operation information satisfies the operation condition (in this case, it indicates up and down movement), the operation determination unit 86 may allow the application to input the operation indicated by the first operation information (for example, the operation of a button or analog stick). This makes it possible to realize a way of playing in which the first user can input operations with the controller 16 only while the information processing device 18 is moving up and down (for example, while the second user is performing a squat).

[0133] The present disclosure has been described above based on the fifth embodiment. The fifth embodiment is illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of each component and each processing process, and that such modifications are also within the scope of the present disclosure.

[0134] <Sixth Embodiment> The sixth embodiment of this disclosure will be described focusing on the differences from the first embodiment. Components of the sixth embodiment that are the same as or equivalent to those of the first embodiment will be denoted by the same reference numerals as those of the first embodiment, and explanations that overlap with the description of the first embodiment will be omitted as appropriate.

[0135] The configuration of the game system 10 in the sixth embodiment is the same as the configuration of the game system 10 in the first embodiment shown in Figure 1. The information processing device 18 is indirectly connected to the game machine 12 via the controller 16. In the game system 10 of the sixth embodiment, the controller 16 determines input information to the application (video game, etc.) executed on the game machine 12 based on both the state or operation input of the controller 16 and the state or operation input of the information processing device 18.

[0136] Figure 14 is a block diagram showing the functional blocks of the controller 16 of the sixth embodiment. The controller 16 comprises a processing unit 120, a storage unit 122, a communication unit 124, and a sensor 126. The processing unit 120 performs various data processing. The storage unit 122 stores data that is referenced or updated by the processing unit 120. The communication unit 124 communicates with external devices according to a predetermined communication protocol. The processing unit 120 sends and receives data with the game machine 12 and the information processing device 18 via the communication unit 124.

[0137] The sensor 126 includes a known sensor (e.g., an IMU) for detecting the state of the controller 16. For example, the sensor 126 may detect the position, orientation, direction of movement, acceleration, angle with respect to a reference direction, angular velocity, distance between the controller 16 and the information processing device 18, etc.

[0138] The processing unit 120 includes a first acquisition unit 130, a second acquisition unit 132, a determination unit 134, and an output unit 136. A computer program implementing the functions of at least some of these multiple functional blocks may be installed in the storage of the controller 16. The processor of the controller 16 may perform the functions of these multiple functional blocks by reading this computer program into main memory and executing it.

[0139] The first acquisition unit 130 acquires first operation information. The first operation information includes either or both information indicating the state of the controller 16 detected by the sensor 126 and information indicating an operation input to an operating member (button, key, analog stick, touchpad, etc.) provided by the controller 16.

[0140] The second acquisition unit 132 acquires second operation information, which is operation information generated and output by the information processing device 18. The second operation information includes information indicating the state of the information processing device 18 or an operation input to the information processing device 18. The state of the information processing device 18 may be detected by a sensor (e.g., an IMU) mounted on the information processing device 18. The second operation information may include operation information, output information, and sensing information output by the information processing device 18 in the first to fourth embodiments.

[0141] The determination unit 134 determines the operation information (hereinafter referred to as "third operation information") to be input to the application executed on the game console 12, based on the first operation information and the second operation information. The third operation information may have the same content as either the first operation information or the second operation information. The output unit 136 inputs the third operation information to the application executed on the game console 12 by transmitting the third operation information determined by the determination unit 134 to the game console 12. According to the game system 10 of the sixth embodiment, a highly engaging way of playing can be realized by coordinating the controller 16 and the information processing device 18.

[0142] The following describes examples (Examples 1-4) of determining input information to an application based on first operation information and second operation information. Examples 1-4 can be combined as appropriate. In each example, the controller 16 may be operated by a first user, and the information processing device 18 may be operated by a second user different from the first user.

[0143] Example 1: The decision unit 134 compares the input time of the first operation information with the input time of the second operation information, and if the difference between the two is less than or equal to a predetermined threshold (e.g., n seconds), it may decide to input (in other words, permit) a third operation information based on at least one of the first operation information and the second operation information into the application. The decision unit 134 may decide to input both the operation indicated by the first operation information and the operation indicated by the second operation information into the application, or it may decide to input only one of the operations into the application. If the difference between the input time of the first operation information and the input time of the second operation information exceeds the threshold, the decision unit 134 may restrict or prohibit inputting information based on at least one of the first operation information and the second operation information into the application.

[0144] The input times to be compared may be the time when the state was detected or the time when the operation was input for both the controller 16 and the information processing device 18, and the input time may be included in both the first operation information and the second operation information. Alternatively, the input time may be the time when both the first operation information and the second operation information were input to the controller 16, or in other words, the time when the controller 16 acquired or received both the first operation information and the second operation information.

[0145] Example 2: The first acquisition unit 130 may further acquire sensor information detected by the sensor 126 mounted on the controller 16. The second acquisition unit 132 may further acquire sensor information detected by the sensor mounted on the information processing device 18. The sensor information may be included in either or both of the first operation information and the second operation information. The determination unit 134, as a detection unit, may detect the distance between the controller 16 and the information processing device 18 based on the sensor information of the controller 16 acquired by the first acquisition unit 130 and the sensor information of the information processing device 18 acquired by the second acquisition unit 132. Known techniques (for example, position measurement using wireless or distance measurement using a ToF camera) may be used to detect the distance.

[0146] The decision unit 134 may change the third operation information to be input to the application based on the first operation information and the second operation information, depending on the distance between the controller 16 and the information processing device 18. For example, if the distance between the controller 16 and the information processing device 18 is less than a predetermined threshold (i.e., close), the decision unit 134 may decide on "press the circle button" as the third operation information based on the first operation information and the second operation information. On the other hand, if the distance between the controller 16 and the information processing device 18 is greater than or equal to the threshold (i.e., far), the decision unit 134 may decide on "press the X button" as the third operation information based on the first operation information and the second operation information.

[0147] In another embodiment, the determination unit 134 may convert the distance between the controller 16 and the information processing device 18 into the time the analog stick of the controller 16 is tilted. For example, if the distance between the controller 16 and the information processing device 18 is less than a predetermined threshold, the operation determination unit 86 may determine a short-duration tilt operation of the analog stick as the third operation information. If the distance between the controller 16 and the information processing device 18 is greater than or equal to the threshold, the operation determination unit 86 may determine a long-duration tilt operation of the analog stick as the third operation information.

[0148] Example 3: The determination unit 134 may determine third operation information to be input to the application based on the first operation information and the second operation information when the state or operation of the controller 16 indicated by the first operation information matches the state or operation of the information processing device 18 indicated by the second operation information. For example, the operation determination unit 86 may determine the rotation angle indicated by the first or second operation information as the third operation information when the rotation angle of the controller 16 indicated by the first operation information matches the rotation angle of the information processing device 18 indicated by the second operation information. In other words, it may permit the input of the rotation angle indicated by the first or second operation information to the application. Note that "matching" includes the difference between the two being less than or equal to a predetermined threshold that can be considered a match.

[0149] The operation determination unit 86 may restrict or prohibit the input of a third operation information based on at least one of the first operation information and the second operation information to the application if the state or operation of the controller 16 indicated by the first operation information does not match the state or operation of the information processing device 18 indicated by the second operation information. For example, if the difference between the rotation angle of the controller 16 indicated by the first operation information and the rotation angle of the information processing device 18 indicated by the second operation information is greater than or equal to a predetermined threshold, the operation determination unit 86 may restrict or prohibit the input of the rotation angle indicated by the first operation information or the second operation information as the third operation information to the application.

[0150] Example 4: The decision unit 134 may decide to input third operation information based on the first operation information into the application when the second operation information indicates a specific state or operation of the predetermined information processing device 18.

[0151] For example, a predetermined specific state or operation of the information processing device 18 (hereinafter also referred to as the "operation condition") may be the information processing device 18 moving up and down. When a second user holds the information processing device 18 and performs a squat, the second operation information will indicate that the information processing device 18 is moving up and down. If the state or operation of the information processing device 18 indicated by the second operation information satisfies the operation condition (in this case, up and down movement), the operation determination unit 86 may allow the operation indicated by the first operation information (for example, the operation of a button or analog stick) to be input to the application as the third operation information. This makes it possible to realize a way of playing in which the first user can input operations with the controller 16 only while the information processing device 18 is moving up and down (for example, while the second user is performing a squat).

[0152] The present disclosure has been described above based on the sixth embodiment. The sixth embodiment is illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of each component and each processing process, and that such modifications are also within the scope of the present disclosure.

[0153] Any combination of the above-described embodiments and modifications is also useful as an embodiment of the present invention. The new embodiments resulting from these combinations possess the combined effects of each of the embodiments and modifications. Furthermore, it will be understood by those skilled in the art that the functions to be performed by each component described in the claims can be achieved by each component shown in the embodiments and modifications individually or in combination thereof.

[0154] The technical concepts described in the above embodiments and modifications can be expressed as follows: [Technology 1] An information processing device that is detachable from the controller of a game machine running an application, comprising a mounting member for attaching the information processing device to the controller, and a circuit configured to the following, wherein the circuit detects the operation or state of an object different from the controller that is attached to the information processing device by the mounting member, and outputs information based on the detection result by the sensor unit. [Technology 2] The information processing device according to Technology 1, wherein the circuit transmits to the game machine operation information that corresponds to the detection result by the sensor unit from among the operation information that can be input from the controller to the game machine. [Technology 3] The information processing device according to Technology 1, wherein the object attached to the information processing device by the mounting member is a sheet-like object on which one or more objects to be operated by a user are drawn, the circuit detects an object on which an operation has been input by a user from among the objects to be operated by a user, and the circuit transmits to the game machine operation information based on the object on which an operation has been input by a user. [Technology 4] The information processing apparatus according to Technology 1, further comprising a storage unit for storing information based on the detection results of the sensor unit. [Technology 5] The information processing apparatus according to Technology 1, further comprising a display unit, wherein the object attached to the information processing apparatus by the mounting member is an ingredient, a dish, or a cooking utensil, the circuit detects the state of the ingredient, the dish, or the cooking utensil, and the circuit causes the display unit to display information based on the state of the ingredient, the dish, or the cooking utensil. [Technology 6] The information processing apparatus according to Technology 1, wherein the object attached to the information processing apparatus by the mounting member is an ingredient, a dish, or a cooking utensil, the circuit detects the state of the ingredient, the dish, or the cooking utensil, and the circuit transmits information based on the state of the ingredient, the dish, or the cooking utensil to the game machine. [Technology 7] The information processing apparatus according to Technology 4, wherein the circuit stores information based on the detected state of the ingredient, the dish, or the cooking utensil.[Technical 8] An information processing device that is detachable from the controller of a game console on which an application is executed, and which includes a mounting member for attaching the information processing device to the controller, the information processing device detects the operation or state of an object different from the controller that is attached to the information processing device by the mounting member, and outputs information based on the result of the detection.

[0155] The technology disclosed herein can be applied to information processing devices, information processing systems, and the like.

[0156] 10 Game system, 12 Game machine, 14 Display device, 16 Controller, 17 Mounting part, 18 Information processing device, 20 Display unit, 21 Mounting member, 22 Clip, 24 Hinge, 26 Protrusion, 30 Sensor, 31 Sensor surface, 32 Processing unit, 34 Storage unit, 36 Communication unit, 40 Conversion rule storage unit, 42 Motion detection unit, 44 Operation information generation unit, 46 Operation information output unit, 47 Mapping unit, 48 Operation target detection unit, 50 Game storage unit, 52 Game execution unit, 54 State detection unit, 56 Output information generation unit, 58 Display control unit, 60 Temperature sensor, 62a, 62b, 62c, 62 Distance sensor, 64 Cooking history storage unit, 66 Information transmission unit, 70 Processing unit, 72 Storage unit, 74 Communication unit, 76 Application storage unit, 80 Application execution unit, 82 Display control unit, 84 Operation information receiving unit, 86 Operation determination unit, 100 Object, 102 Content, 110 Paper, 112a, 112b, 112c Buttons, 114 Slider, 116 Knob, 120 Processing unit, 122 Storage unit, 124 Communication unit, 126 Sensor, 130 First acquisition unit, 132 Second acquisition unit, 134 Determination unit, 136 Output unit.

Claims

1. An information processing device that is detachable from the controller of a game console on which an application is executed, comprising: a mounting member for attaching the information processing device to the controller; a sensor unit attached to the information processing device by the mounting member for detecting the operation or state of an object different from the controller; and an output unit that outputs information based on the detection result by the sensor unit.

2. The information processing apparatus according to claim 1, wherein the output unit transmits to the game machine operation information corresponding to the detection result by the sensor unit from among the operation information that can be input from the controller to the game machine.

3. The information processing apparatus according to claim 1, wherein the object attached to the information processing apparatus by the mounting member is a sheet-like object on which one or more objects to be operated by a user are drawn, the sensor unit detects an object on which an operation has been input by a user from among the objects on which an operation has been input by a user, and the output unit transmits operation information based on the object on which an operation has been input by a user.

4. The information processing apparatus according to any one of claims 1 to 3, further comprising a storage unit for storing information based on the detection results from the sensor unit.

5. The information processing apparatus according to claim 1, further comprising a display unit, wherein the object attached to the information processing apparatus by the mounting member is an ingredient, a dish, or a cooking utensil, the sensor unit detects the state of the ingredient, the dish, or the cooking utensil, and the output unit causes the display unit to display information based on the state of the ingredient, the dish, or the cooking utensil.

6. The information processing apparatus according to claim 1, wherein the object attached to the information processing apparatus by the mounting member is food, food, or cooking utensils, the sensor unit detects the state of the food, food, or cooking utensils, and the output unit transmits information based on the state of the food, food, or cooking utensils to the game machine.

7. The information processing apparatus according to claim 5 or 6, further comprising a storage unit for storing information based on the state of the food, the dish, or the cooking utensil detected by the sensor unit.

8. An information processing device that is detachable from the controller of a game console on which an application is executed, and which includes a mounting member for attaching the information processing device to the controller, the information processing device detects the operation or state of an object different from the controller that is attached to the information processing device by the mounting member, and outputs information based on the result of the detection.

Images