Voice user interface program, recording medium, and voice user interface processing method

Abstract

To permit a natural communication between a player and a game character.SOLUTION: A voice user interfaced program allows a control part 7 of a head mount display 3 to function as: a first utterance processing part 25 for determining whether or not one portion of a first part of a preset word is uttered; a first action execution processing part 27 for executing first processing corresponding to a word before the word is produced to the end when it is determined that one portion of the first part of the word is produced; a second utterance processing part 29 for determining whether or not a player produces the word to the end in parallel with execution of the first processing; and a second action execution processing part 31 for executing second processing on the basis of a determination result regarding whether or not the word is produced to the end.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an audio user interface program, a recording medium, and an audio user interface processing method.

Background Art

[0002] Conventionally, a game machine equipped with a voice input type human interface has been known. For example, in Patent Document 1, when a player's utterance is voice-recognized, the linguistic meaning thereof is reflected in the next behavior of the dialogue partner character on the game video, and it is possible for the player in the real world and the character in the virtual community in the game world to communicate with each other. A game machine is described.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above prior art, the voice uttered by the player is recognized as a word, and an action corresponding to the content of the recognized word is performed on the character. For this reason, for example, in the case of communication in which voice and action are performed in parallel, the action of the character may be delayed, resulting in unnatural communication.

[0005] The present invention has been made in view of such problems, and an object thereof is to provide an audio user interface program, a recording medium, and an audio user interface processing method capable of achieving natural communication between a player and a game character.

Means for Solving the Problems

[0006] To achieve the above object, the voice user interface program of the present invention causes an information processing apparatus to function as a first voice determination processing unit that determines whether a first part of a preset word has been uttered by a player, and a first processing execution processing unit that, when it is determined that the first part of the word has been uttered, executes a first process corresponding to the word before the word is uttered to the end. In parallel with the execution of the first process, a second voice determination processing unit that determines whether the word has been uttered to the end by the player, and a second processing execution processing unit that executes a second process based on the determination result of whether the word has been uttered to the end.

[0007] To achieve the above object, the recording medium of the present invention is a recording medium readable by an information processing apparatus, on which the above voice user interface program is recorded.

[0008] To achieve the above object, the voice user interface processing method of the present invention is a game processing method executed by an information processing apparatus, including a step of determining whether a first part of a preset word has been uttered by a player, a step of executing a first process corresponding to the word before the word is uttered to the end when it is determined that the first part of the word has been uttered, a step of determining whether the word has been uttered to the end by the player in parallel with the execution of the first process, and a step of executing a second process based on the determination result of whether the word has been uttered to the end.

Advantages of the Invention

[0009] According to the voice user interface program of the present invention and the like, natural communication can be achieved between the player and the game character.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0012] <1. Configuration of the game system> First, with reference to FIGS. 1 and 2, an example of the configuration of the game system 1 according to the embodiment will be described. As shown in FIG. 1, the game system 1 includes a head-mounted display 3. Note that in addition to the head-mounted display 3, the game system 1 may include a game machine main body, a game controller for the player to operate, and the like.

[0013] The head-mounted display 3 is a display device that can be worn on the player's head or face and realizes so-called MR (Mixed Reality). The head-mounted display 3 has a transmissive display unit 5, and superimposes and displays a virtual image related to the game generated by the control unit 7 (see FIG. 2) on the image of the real space.

[0014] As shown in FIG. 2, the head-mounted display 3 includes a display unit 5, a head direction detection unit 9, a position detection unit 11, an audio input unit 13, an audio output unit 15, a hand movement detection unit 17, and a control unit 7.

[0015] The display unit 5 is composed of, for example, a transmissive (see-through) liquid crystal display, an organic EL display, or the like, and superimposes and displays a virtual image related to the game generated by the control unit 7 on the image of the real space that can be seen through, for example, as a holographic video. The virtual image may be either a 2D image or a 3D image, and may be either a still image or a moving image. Note that the display unit 5 may be non-transmissive, and a virtual image generated by the control unit 7 may be superimposed and displayed on an image of the real space captured by, for example, a camera.

[0016] The head direction detection unit 9 detects the angle, angular velocity, or angular acceleration of the player's head, etc., and detects the orientation of the head (the orientation of the face) based on the detection result. The orientation of the head may be detected as a direction (vector) in the stationary coordinate system of the real space generated by the space recognition process that recognizes the real space around the player using, for example, a depth sensor or a camera. The method for detecting the direction of the head is not particularly limited, and various detection methods can be adopted. For example, an acceleration sensor, a gyro sensor, etc. may be provided in the head-mounted display 3, and based on the detection results of these sensors, the control unit 7 may calculate the direction of the player's head.

[0017] The position detection unit 11 detects the position of the player's head. The method for detecting the position of the head is not particularly limited, and various detection methods can be adopted. For example, cameras and depth sensors may be provided at multiple locations around the head-mounted display 3, the depth sensor is used to recognize the space around the player (the real space), and based on the detection results of the multiple cameras, the control unit 7 may recognize the position of the player's head in the surrounding space. Also, for example, a camera may be installed outside the head-mounted display 3, and a landmark such as a light-emitting unit may be installed in the head-mounted display 3, and the position of the player's head may be detected by the external camera.

[0018] The voice input unit 13 is composed of, for example, a microphone, etc., and inputs the voice emitted by the player or other external sounds. The input voice of the player is recognized as words by the control unit 7 through voice recognition processing, and a predetermined process is executed based on the recognized words.

[0019] The voice output unit 15 is composed of, for example, a speaker, etc., and outputs voice to the player's ears. For example, the voice emitted by the character, sound effects, BGM, etc. are output.

[0020] The hand movement detection unit 17 is composed of, for example, a camera, an infrared sensor, etc., and detects the shape and movement of the player's hand as a hand action. The control unit 7 executes predetermined processing based on the detected hand action.

[0021] The control unit 7 executes various processes based on the detection signals of various sensors and voice inputs. The various processes are, for example, image display processing, position detection processing, space recognition processing, voice recognition processing, voice output processing, hand action detection processing, etc. In addition, it may be possible to execute a wide variety of other processes. The control unit 7 generates or changes a virtual image to be displayed on the display unit 5 based on the processing results of the head direction detection unit 9, position detection unit 11, voice input unit 13, voice output unit 15, hand movement detection unit 17, etc. of the head-mounted display 3, and expresses MR (mixed reality).

[0022] <2. Outline of the game> In this embodiment, a case where the control unit 7 executes a game program, which is an example of a voice user interface program, and a game processing method, which is an example of a voice user interface processing method, will be described. Next, an example of the outline of a game provided by the execution of the game program and game processing method of this embodiment by the control unit 7 will be described.

[0023] The game according to this embodiment enables communication with a game character that appears to exist in the real space for the player by superimposing an image of a virtual game character on an image of the real space. The actions and behaviors of the game character change according to various operation inputs (head movements, hand movements, voice inputs, etc.) by the player. The type of the game character is not particularly limited, but is typically a human male character or a human female character. Note that the game character may be a non-human animal character, a virtual creature character other than humans and animals, or a character such as a robot or an object (so-called object) other than a creature.

[0024] FIG. 3 shows an example of a game screen. In this example, a female game character 19 is displayed superimposed on an image 21 of the player's room, which is, for example, the real space.

[0025] In this embodiment, the player and the game character 19 execute communication in which at least part of the voice and the action are performed in parallel. Hereinafter, as an example of such communication, the details of the processing will be described for the case where the player and the game character 19 execute a game in which they compete for victory or defeat, such as "Turn around, whee" or "Rock-paper-scissors".

[0026] <3. Functional Configuration of Control Unit> An example of the functional configuration of the control unit 7 of the head-mounted display 3 will be described with reference to FIG. 4.

[0027] As shown in FIG. 4, the control unit 7 (an example of an information processing apparatus) includes a voice recognition processing unit 23, a first voice determination processing unit 25, a first action execution processing unit 27, a second voice determination processing unit 29, a second action execution processing unit 31, an action detection processing unit 33, and a third action execution processing unit 35.

[0028] The voice recognition processing unit 23 converts the voice of the player input by the voice input unit 13 into a corresponding text (character string). Specifically, for example, the voice is analyzed by frequency analysis or the like, phonemes are recognized using a voice recognition dictionary (acoustic model, language model, pronunciation dictionary, etc.), and converted into text. Note that techniques such as machine learning and deep learning may be used for the voice recognition processing.

[0029] The first voice determination processing unit 25 determines whether a first part of a preset word has been uttered by the player based on the text converted by the speech recognition processing unit 23. The "preset word" is not particularly limited as long as it represents a word for communication in which at least part of the voice and action are performed simultaneously between the player and the game character. For example, it may be a word representing a game in which the player and the game character compete for victory or defeat. Specifically, it may be "Look over there, whee" or "Jankenpon", etc. The "first part" is, for example, "Look over there..." or "Look over there, whee..." when the word is "Look over there, whee". When the word is "Jankenpon", it is, for example, "Jan..." or "Janken...", etc. Note that the above is an example, and it may be separated at a part different from the above and used as the first part.

[0030] In addition, when there are variations in the expression of the preset word according to the region, age, etc., the word may be set so as to include them.

[0031] When the first voice determination processing unit 25 determines that a first part of a word has been uttered, the first action execution processing unit 27 (an example of the first processing execution processing unit) executes a first process corresponding to the word before the word is fully uttered. Specifically, as the "first process", the first action execution processing unit 27 causes the game character to start a first action corresponding to the word. The "first action corresponding to the word" is the action of the game when the word represents a game, for example. For example, in the case of "Turn around, ho", the preparatory actions corresponding to the part "Turn around" (for example, actions such as swaying the face or body to keep rhythm, or actions waiting for the player to point, etc. It may also be an action of starting to move the face in any direction), and the action of turning the face in any of the up, down, left, or right directions corresponding to the part "ho". Therefore, the first action execution processing unit 27 causes the game character to start the above preparatory actions at the timing when, for example, "that way" is uttered by the player. Note that the direction in which the face is turned may be determined randomly, for example, or may be determined by reflecting the personality or ability of the game character, etc.

[0032] The first action is composed of the above preparatory actions and the action of moving the face, but the switching timing may be set to switch at a fixed timing corresponding to the general utterance speed of, for example, "Turn around, ho". In this case, the processing can be simplified and the processing load can be reduced. Also, for example, it may be set to switch from the preparatory action to the action of moving the face at the timing when it is detected that "ho" in "ho" has been uttered. In this case, for example, it can also handle the case where the player stretches the voice as "Turn around~" and then utters "ho", and the voice of the player and the actions of the game character can be accurately synchronized.

[0033] Also, for example, in the case of "Jankenpon", it includes a preparatory action corresponding to the "Janken" part (such as an action of swaying the hands or arms to keep rhythm, or an action of waiting for the player to put out their hand. It may also be an action of starting to form any shape with the hand), and an action of putting out the hand in the form of either rock, scissors, or paper corresponding to the "pon" part. Therefore, the first action execution processing unit 27 causes the game character to start the above preparatory action at the timing when, for example, the player utters "Jan". Note that the shape of the hand may be determined randomly, for example, or may be determined by reflecting the personality or ability of the game character, etc.

[0034] The first action is composed of the above preparatory action and the action of forming and putting out the shape of rock, scissors, or paper with the hand. The switching timing may be set to switch at a fixed timing corresponding to, for example, the general speaking speed of "Jankenpon". In this case, the processing can be simplified and the processing load can be reduced. Also, for example, it may be set to switch from the preparatory action to the action of forming and putting out the shape with the hand at the timing when it is detected that "Po" of "Pon" has been uttered. In this case, it can also handle, for example, the case where the player elongates their voice as "Jaaanken" and then utters "Pon", and the voice of the player and the actions of the game character can be accurately synchronized.

[0035] The second voice determination processing unit 29 determines whether or not the player has uttered the words to the end in parallel with the execution of the first action by the first action execution processing unit 27 based on the text converted by the above voice recognition processing unit 23. That is, the first voice determination processing unit 25 determines whether or not a part of the beginning of the words has been uttered, while the second voice determination processing unit 29 determines whether or not all of the words have been uttered by the player. The determination by the second voice determination processing unit 29 continues even after the determination by the first voice determination processing unit 25 is satisfied and the game character starts the first action, and is executed in parallel with the first action.

[0036] For example, when the word represents a game, the second voice determination processing unit 29 determines whether the word representing the game has been uttered to the end by the player in parallel with the execution of the game action. For example, when the word is "Turn around, whee", the second voice determination processing unit 29 determines whether the word "Turn around, whee" has been uttered to the end by the player in parallel with the execution of the above-described preparatory action and the action of moving the face of "Turn around, whee". For example, when the word is "Jankenpon", the second voice determination processing unit 29 determines whether the word "Jankenpon" has been uttered to the end by the player in parallel with the execution of the above-described preparatory action and the action of holding out the hand of "Jankenpon".

[0037] The second action execution processing unit 31 (an example of the second processing execution processing unit) executes a second process based on the determination result of whether the word has been uttered to the end by the second voice determination processing unit 29. Specifically, when the second voice determination processing unit 29 determines that the word has not been uttered to the end, the second action execution processing unit 31 causes the game character to execute a second action different from the first action as the "second process". "The word has not been uttered to the end" includes cases where the player stops uttering in the middle of the word or utters a word different from the set word in the remaining part of the word. The "second action" is, for example, an action representing the reaction of the game character to the player not uttering a predetermined word to the end. The second action execution processing unit 31 may cause the game character to execute an action of getting angry with the player as the second action. Note that not limited to the action of getting angry, an action expressing other emotions (for example, laughing, sulking, grieving, etc.) may be executed.

[0038] The action detection processing unit 33 detects the player's actions. For example, based on the shape and movement of the player's hand detected by the manual operation detection unit 17, the action detection processing unit 33 detects in which direction (an example of an action) the player's finger is pointing, whether it is up, down, left, or right, or in which shape of rock, scissors, or paper (an example of an action) the player's hand is formed, etc. Further, based on the angle, angular velocity, or angular acceleration of the player's head detected by the head direction detection unit 9, the action detection processing unit 33 detects in which direction (an example of an action) the player's face is facing, whether it is up, down, left, or right.

[0039] The third action execution processing unit 35 (an example of the second processing execution processing unit) executes a second process based on the determination result of whether the word has been fully pronounced by the second voice determination processing unit 29. Specifically, when the second voice determination processing unit 29 determines that the word has been fully pronounced, the third action execution processing unit 35 determines a third action based on the content of the first action executed by the game character and the content of the player's action detected by the action detection processing unit 33 as the "second process", and causes the game character to execute the third action. For example, when executing a game in which the player and the game character compete for victory or defeat, the third action execution processing unit 35 determines victory or defeat based on the content of the action of the game executed by the game character and the content of the player's action detected, and may cause the game character to execute a third action corresponding to the result of the victory or defeat.

[0040] For example, when the player and the game character play the "Turn around, whee!" game, the third action execution processing unit 35 determines the winner or loser based on the direction of the face due to the action executed by the game character and the direction of the finger due to the detected action of the player, and executes a third action corresponding to the result of the win or loss. For example, when the direction of the face and the direction of the finger match and the game character loses, an action of showing frustration may be executed as the third action. Also, for example, when the direction of the face and the direction of the finger do not match, an action of proceeding to the next rock-paper-scissors may be executed as the third action.

[0041] For example, when the player and the game character play the "Rock, paper, scissors" game, the third action execution processing unit 35 determines the winner or loser based on the hand shape due to the action executed by the game character and the hand shape due to the detected action of the player, and executes a third action corresponding to the result of the win or loss. For example, when the game character wins, an action of showing joy may be executed as the third action. Also, for example, when the game character loses, an action of showing frustration may be executed as the third action. Also, for example, when it is a draw, an action of proceeding to the next rock-paper-scissors (e.g., "It's a draw, right!") may be executed as the third action.

[0042] Note that the processing and the like in each processing unit described above are not limited to the examples of the division of these processes. For example, they may be processed by an even smaller number of processing units (e.g., one processing unit), or may be processed by further subdivided processing units. Also, the functions of each processing unit described above are implemented by a game program executed by the CPU 301 (see FIG. 14 described later), but for example, a part of them may be implemented by an actual device such as an ASIC, an FPGA, or other electric circuits.

[0043] <4. Specific examples of the game screen> Using FIGS. 5 to 9, a specific example of the game screen displayed on the display unit 5 of the head-mounted display 3 will be described.

[0044] FIG. 5 shows an example of the game screen when the direction of the player's finger and the direction of the face of the game character 19 do not match in "Look over there, whee". Note that the player's voice 37 is shown in a speech bubble to the side of the game screen.

[0045] FIG. 5(a) shows the state of the game character 19 before the player starts speaking "Look over there, whee" or before the word "over there" has finished being spoken after the start of speaking. At this point, the game character 19 has not started the action related to the game of "Look over there, whee".

[0046] FIG. 5(b) shows the state of the game character 19 when the player starts speaking "Look over there, whee" and has finished speaking the first part, "over there". At this point, the game character 19 starts the preparatory action described above as the first action. In the example shown in FIG. 5, the game character 19 is performing an action (an example of the first action) of waiting for the player's pointing finger while being excited, for example, with a pounding heart and a thrill. This state continues while the word "Look" is being spoken.

[0047] FIG. 5(c) shows the state of the game character 19 when the player speaks "whee". The game character 19 executes an action of moving its face in one of the directions of up, down, left, or right. In the example shown in FIG. 5, the game character 19 moves its face to the right as seen from the player, and the player is pointing the finger 39 to the left. In this case, since the direction of the face and the direction of the finger do not match, the game character 19 then executes an action of proceeding to the next rock-paper-scissors (an example of the third action).

[0048] FIG. 6 shows an example of the game screen when the direction of the player's finger and the direction of the face of the game character 19 match in "Look over there, whee".

[0049] Since FIGS. 6(a) and 6(b) are the same as FIGS. 5(a) and 5(b) described above, the description thereof is omitted.

[0050] FIG. 6(c) shows the state of the game character 19 when the player utters "hoi". In the example shown in FIG. 6, the game character 19 moves its face to the left as seen from the player, and the player points the finger 39 to the left. In this case, since the direction of the face and the direction of the finger match, and the defeat of the game character 19 is determined, as shown in FIG. 6(d), the game character 19 executes an action corresponding to defeat (an example of the third action).

[0051] FIG. 7 shows an example of a game screen when the player does not utter words until the end in "turn your face that way and say hoi".

[0052] Since FIGS. 7(a) and 7(b) are the same as FIGS. 5(a) and 5(b) described above, the description thereof is omitted.

[0053] FIG. 7(c) shows the state of the game character 19 when the player utters only "turn your face that way" and then does not utter "hoi". The example shown in FIG. 7(c) is a case where the timing to switch from the preparation motion to the face-moving motion is fixedly set, and the game character 19 has executed, for example, the motion of moving the face to the right as seen from the player. In this case, since the player has not uttered "turn your face that way and say hoi" until the end, as shown in FIG. 7(d), the game character 19 executes an action of getting angry with the player (an example of the second action).

[0054] Note that, for example, when switching from the preparation motion to the face-moving motion at the timing when it is detected that "ho" in "hoi" has been uttered, since "hoi" has not been uttered, the switch to the face-moving motion does not occur. In this case, the game character 19 may directly execute the action in FIG. 7(d) from the state in FIG. 7(b) without executing the action in FIG. 7(c) above.

[0055] FIG. 8 shows an example of a game screen when a player utters words until the end in "Jankenpon".

[0056] FIG. 8(a) shows the state of the game character 19 before the player starts uttering the words of "Jankenpon" or after starting to utter the words and before finishing uttering "Jan". At this point, the game character 19 has not started the action related to the game of "Jankenpon".

[0057] FIG. 8(b) shows the state of the game character 19 when the player starts uttering the words of "Jankenpon" and finishes uttering "Jan" which is the first part of it. At this point, the game character 19 starts the preparatory action described above as the first action. In the example shown in FIG. 8, the game character 19 is performing an action of swaying the hand up and down to keep rhythm (an example of the first action). This state continues while "Ken" is being uttered.

[0058] FIG. 8(c) shows the state of the game character 19 when the player utters "Pon". The game character 19 executes an action of putting out the hand 41 in the form of either rock, scissors, or paper. In the example shown in FIG. 8, the game character 19 puts out the hand 41 in the form of scissors, and the player puts out the hand 43 in the form of paper. In this case, since the victory of the game character 19 is determined, the game character 19 may then execute an action corresponding to victory, such as being happy (an example of the third action). Alternatively, when the game of "Achi muite hoi" is being executed, an action of shouting "Achi muite hoi" and pointing with a finger (an example of the third action) may be executed.

[0059] FIG. 9 shows an example of a game screen when a player does not utter words until the end in "Jankenpon".

[0060] FIGS. 9(a) and 9(b) are the same as FIGS. 8(a) and 8(b) described above, so the description is omitted.

[0061] Figure 9(c) shows the state of the game character 19 when the player only utters "rock-paper-scissors" and then does not utter "pon". The example shown in Figure 9(c) is a case where the timing to switch from the preparatory motion to the motion of making and showing the shape of rock-paper-scissors with the hand is fixedly set, and the game character 19 has executed the motion of, for example, showing the hand 41 in the shape of scissors. In this case, since the player has not uttered "rock-paper-scissors-pon" until the end, as shown in Figure 9(d), the game character 19 executes an action of getting angry with the player (an example of the second action).

[0062] Incidentally, for example, in the case of switching from the preparatory motion to the motion of making and showing the shape of rock-paper-scissors with the hand at the timing when it is detected that "po" of "pon" has been uttered, since "pon" has not been uttered, the motion of showing the hand is not switched. In this case, the game character 19 may directly execute the action in Figure 9(d) from the state in Figure 9(b) without executing the action in Figure 9(c) above.

[0063] <5. Processing Procedure Executed by the Control Unit> Next, an example of the processing procedure executed by the control unit 7 will be described with reference to FIGS. 10 and 11.

[0064] As shown in FIG. 10, in step S100, the control unit 7 executes a rock-paper-scissors process in which the player and the game character 19 play the game of "rock-paper-scissors-pon". Details of the rock-paper-scissors process will be described later (see FIG. 11).

[0065] In step S5, the control unit 7 determines whether the player has won in the rock-paper-scissors process in step S100 above. If the player has won (step S5: YES), the process proceeds to the next step S10.

[0066] In step S10, the control unit 7 determines, by the first voice determination processing unit 25, whether "that way" which is a part of the beginning of "Turn that way, whee" has been uttered by the player. This step S10 is repeated until "that way" is uttered (step S10: NO), and when "that way" is uttered (step S10: YES), the process proceeds to the next step S15.

[0067] In step S15, the control unit 7 causes, by the first action execution processing unit 27, the game character 19 to start an action corresponding to the game of "Turn that way, whee" before "Turn that way, whee" is uttered to the end. The said action is constituted by, for example, the above-described preparatory action and the action of moving the face in any of the up, down, left, or right directions.

[0068] In step S20, the control unit 7 recognizes, by the second voice determination processing unit 29, the voice uttered by the player in parallel with the execution of the action by the game character 19 started in the above step S15.

[0069] In step S25, the control unit 7 determines, by the second voice determination processing unit 29, whether the player has uttered "Turn that way, whee" to the end. If "Turn that way, whee" has not been uttered to the end (step S25: NO), the process proceeds to step S30.

[0070] In step S30, the control unit 7 causes, by the second action execution processing unit 31, the game character 19 to execute an action of getting angry with the player. Then, the process proceeds to step S80 described later.

[0071] On the other hand, in the above step S25, when "Turn that way, whee" has been uttered to the end (step S25: YES), the process proceeds to step S35.

[0072] In step S35, the control unit 7 detects, by the action detection processing unit 33, the hand action of the player (which direction the finger 39 is pointing in the up, down, left, or right directions).

[0073] In step S40, the control unit 7 determines, based on the content of the action executed by the game character 19 by the third action execution processing unit 35 and the player's hand action detected in step S35, whether the direction of the player's finger and the direction of the face of the game character 19 match. If they do not match (step S40: NO), the process returns to the first step S100. On the other hand, if they match (step S40: YES), the process proceeds to the next step S45.

[0074] In step S45, the control unit 7 determines the player's victory by the third action execution processing unit 35.

[0075] In step S50, the control unit 7 causes the third action execution processing unit 35 to execute an action corresponding to losing, such as showing frustration, on the game character 19. Then, the process proceeds to step S80 described later.

[0076] In step S5 above, if it is determined that the game character 19 won in the rock-paper-scissors process of step S100 (step S5: NO), the process proceeds to the next step S55.

[0077] In step S55, the control unit 7 causes the game character 19 to execute an action of pointing in one of the up, down, left, or right directions with a finger along with the shout "Turn around, hey".

[0078] In step S60, the control unit 7 detects, by the action detection processing unit 33, in which direction the player's face is facing, i.e., up, down, left, or right.

[0079] In step S65, the control unit 7 determines whether the direction of the finger of the game character 19 matches the direction of the player's face based on the content of the action executed by the game character 19 and the direction of the player's face detected in step S60 above. If they do not match (step S65: NO), it returns to the first step S100. On the other hand, if they match (step S65: YES), it moves to the next step S70.

[0080] In step S70, the control unit 7 determines the victory of the game character 19.

[0081] In step S75, the control unit 7 causes the game character 19 to execute an action corresponding to winning, such as being happy. Then, it moves to step S80.

[0082] In step S80, the control unit 7 determines whether to execute the game of "Turn around, ho!" again. If the player executes a predetermined re-execution operation or the like and the game of "Turn around, ho!" is to be executed again (step S80: YES), it returns to the first step S100. On the other hand, if the player executes a predetermined end operation or the like and the game of "Turn around, ho!" is to be ended (step S80: NO), this flowchart ends.

[0083] FIG. 11 shows an example of the detailed procedure of the rock-paper-scissors process in step S100 described above.

[0084] As shown in FIG. 11, in step S110, the control unit 7 determines, by the first voice determination processing unit 25, whether "jan", which is a part of the beginning of "rock-paper-scissors", has been uttered by the player. This step S110 is repeated until "jan" is uttered (step S110: NO), and when "jan" is uttered (step S110: YES), it moves to the next step S120.

[0085] In step S120, before "Jankenpon" is completely uttered by the first action execution processing unit 27, an action corresponding to the "Jankenpon" game is started for the game character 19. The action is composed of, for example, the above-described preparatory action and an action of making and showing either the hand gesture of rock-paper-scissors.

[0086] In step S130, the control unit 7, by the second voice determination processing unit 29, recognizes the voice uttered by the player in parallel with the execution of the action by the game character 19 started in the above step S120.

[0087] In step S140, the control unit 7, by the second voice determination processing unit 29, determines whether or not the player has uttered "Jankenpon" to the end. If "Jankenpon" has not been uttered to the end (step S140: NO), the process proceeds to step S150.

[0088] In step S150, the control unit 7, by the second action execution processing unit 31, causes the game character 19 to execute an action of getting angry with the player. Then, the process proceeds to step S80 in FIG. 10.

[0089] On the other hand, in the above step S140, if "Jankenpon" has been uttered to the end (step S140: YES), the process proceeds to step S160.

[0090] In step S160, the control unit 7, by the action detection processing unit 33, detects the player's hand action (whether the shape of the hand 43 is in the form of rock, scissors, or paper).

[0091] In step S170, the control unit 7, by the third action execution processing unit 35, determines the win or loss based on the hand shape of the action executed by the game character 19 and the hand shape of the player's hand action detected in the above step S160.

[0092] In step S180, the control unit 7 determines whether the determination of victory or defeat is a draw. If it is a draw (step S180: YES), it returns to the first step S110. On the other hand, if it is not a draw (step S180: NO), this routine ends and it moves to step S5 in FIG. 10.

[0093] Note that the above-described processing procedure is an example, and at least a part of the above procedure may be deleted or changed, or procedures other than the above may be added. Also, the order of at least a part of the above procedure may be changed, or a plurality of procedures may be combined into a single procedure.

[0094] <6. Effects of the Embodiment> As described above, the game program (an example of an audio user interface program) of the present embodiment causes the control unit 7 of the head-mounted display 3 to function as a first voice determination processing unit 25 that determines whether a first part of a preset word has been uttered by the player, a first action execution processing unit 27 that, when it is determined that the first part of the word has been uttered, executes a first process corresponding to the word before the word is uttered to the end, a second voice determination processing unit 29 that determines whether the word has been uttered to the end by the player in parallel with the execution of the first process, and a second action execution processing unit 31 that executes a second process based on the determination result of whether the word has been uttered to the end.

[0095] Also, in the present embodiment, when the first action execution processing unit 27 determines that the first part of the word has been uttered, before the word is uttered to the end, as a first process, it causes the game character to start a first action corresponding to the word, and when the second action execution processing unit 31 determines that the word has not been uttered to the end, as a second process, it may cause the game character to execute a second action different from the first action.

[0096] In general, in a game system equipped with a voice input function, the voice uttered by the player is recognized as a word, and an action corresponding to the content of the recognized word is performed on the game character, thereby enabling communication between the player and the game character. For this reason, it is necessary to wait for the player to finish speaking. However, in the case of communication where, for example, speaking and actions are performed simultaneously, the actions of the game character may be delayed, resulting in unnatural communication in some cases.

[0097] In the game program of this embodiment, when a player utters the first part of a preset word, before the word is uttered to the end, a first action corresponding to the word is started on game character 19. Thereby, an action corresponding to the content of the word assumed at the timing when the player utters the first part of the word can be started on game character 19. In this way, before the player finishes speaking the word, the action corresponding to the word can be immediately started on game character 19, so that an action can be executed on game character 19 simultaneously with the player's speech. Therefore, it is possible to suppress a delay in the actions of game character 19.

[0098] On the other hand, it is also conceivable that the player does not utter the preset word to the end, for example, the player stops speaking in the middle of the word or utters a word different from the preset word in the remaining part of the word. In such a case, by having game character 19 execute a second action different from the first action, it is possible to recover so that performing the first action does not seem unnatural. As described above, natural communication that achieves both real-time performance and interactivity can be achieved between the player and game character 19.

[0099] Also, by performing a first action promptly and adding a second action for recovery if the player does not ultimately utter all the words, complex processes such as finely dividing the voice to perform voice recognition processing so that, for example, the uttered content of the player and the action content of the game character 19 do not differ, or checking the consistency for each divided word, become unnecessary. Thus, the processing burden can be reduced and the processing speed can be improved.

[0100] Also, in this embodiment, the control unit 7 may be further configured to function as an action detection processing unit 33 that detects the player's action, and a third action execution processing unit 35 that determines a third action based on the content of the first action executed by the game character 19 and the content of the detected player's action when it is determined that the words have been uttered to the end, and causes the game character 19 to execute the third action.

[0101] In this case, the next action can be determined taking into account the content of the action executed by the game character 19 and the content of the player's action, and the game character 19 can be caused to execute it. As a result, when the player utters the preset words to the end, natural communication between the player and the game character 19 can be smoothly continued without intervening voice error processing such as the second action.

[0102] Also, in this embodiment, the first voice determination processing unit 25 determines whether the player has uttered the first part of the words representing the game in which the player and the game character 19 compete for victory or defeat. When the first action execution processing unit 27 determines that the first part of the words has been uttered, before the words are uttered to the end, it causes the game character to start the action of the game. The second voice determination processing unit 29 determines whether the player has uttered the words to the end in parallel with the execution of the action of the game. When the third action execution processing unit 35 determines that the words have been uttered to the end, it determines the victory or defeat based on the content of the action of the game executed by the game character 19 and the content of the detected action of the player, and may cause the game character 19 to execute a third action corresponding to the result of the victory or defeat.

[0103] In this case, a game in which the player and the game character 19 compete for victory or defeat can be executed in real time and interactively.

[0104] Also, in this embodiment, the second action execution processing unit 31 may cause the game character 19 to execute an action of getting angry with the player as the second action.

[0105] In this case, for example, when the player stops speaking in the middle of the words or utters words different from the assumed words in the remaining part of the words, the game character 19 can be made angry. Thereby, the reality of the communication between the player and the game character 19 can be improved.

[0106] Also, in the present embodiment, the first voice determination processing unit 25 determines whether the first part of the phrase "Look over there, hey" has been uttered by the player. When the first action execution processing unit 27 determines that the first part of "Look over there, hey" has been uttered, before "Look over there, hey" is uttered to the end, it causes the game character 19 to start the action of the "Look over there, hey" game. The second voice determination processing unit 29 determines whether the player has uttered the phrase "Look over there, hey" to the end in parallel with the execution of the action of the "Look over there, hey" game. When the second action execution processing unit 31 determines that the phrase "Look over there, hey" has not been uttered to the end, it causes the game character 19 to execute the second action. When the third action execution processing unit 35 determines that the phrase "Look over there, hey" has been uttered to the end, it determines the win or loss based on the direction of the face by the action executed on the game character 19 and the direction of the finger by the action of the detected player, and may cause the game character 19 to execute the third action corresponding to the result of the win or loss.

[0107] In this case, the "Look over there, hey" game can be executed in real time and interactively between the player and the game character 19.

[0108] Also, in the present embodiment, the first voice determination processing unit 25 determines whether the first part of the phrase "Jankenpon" has been uttered by the player. When the first action execution processing unit 27 determines that the first part of "Jankenpon" has been uttered, before "Jankenpon" is uttered to the end, it causes the game character 19 to start the action of the "Jankenpon" game. The second voice determination processing unit 29 determines whether the player has uttered the phrase "Jankenpon" to the end in parallel with the execution of the action of the "Jankenpon" game. When the second action execution processing unit 31 determines that the phrase "Jankenpon" has not been uttered to the end, it causes the game character 19 to execute the second action. When the third action execution processing unit 35 determines that the phrase "Jankenpon" has been uttered to the end, it determines the win or loss based on the hand shape by the action executed by the game character 19 and the hand shape by the detected action of the player, and may cause the game character 19 to execute the third action corresponding to the result of the win or loss.

[0109] In this case, the "Janken" game can be executed in real time and interactively between the player and the game character. <7. Modification Example>

[0110] Note that the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit and technical idea thereof.

[0111] For example, in the above, the case where communication is carried out by playing games such as "Turn around, ho!" or "Jankenpon" between the player and the game character has been described. However, as long as it is communication in which at least a part of the voice and the action are performed simultaneously, the type thereof is not limited. In addition to the above, for example, a game such as "One, two, three, shoot!" in which each of the player and the game character can compete for victory or defeat by raising zero, one, or two thumbs and arguing about the total number (0 to 4) may be executed. In this case, for example, when "One" which is a part of the beginning of "One, two, three, shoot!" is uttered, before "One, two, three, shoot!" is uttered to the end, the action of the game of "One, two, three, shoot!" may be started for the game character 19.

[0112] Also, in the above, the case where the player and the game character communicate one-on-one has been described. However, for example, when playing games such as "Jankenpon" or "One, two, three, shoot!", at least one of the player or the game character may be plural. When there are a plurality of players, the control units 7 of the head-mounted displays 3 of the respective players communicate with each other, and by sharing the detection results of the hand actions of the respective players, the victory or defeat of the game character and each player may be determined respectively. When there are a plurality of game characters, each game character may be independently controlled and each action may be executed individually for each game character.

[0113] Also, in the above, the case where the player plays the game while wearing the head-mounted display 3 which is a display device for realizing so-called MR has been described. However, as long as it is a game machine equipped with a voice input function and a hand action detection function, the type of the game machine is not limited to the head-mounted display. For example, as shown in FIG. 12, a game system 1A having an information processing device 45, a game controller 47, a display device 49, a microphone 51, a camera 53, etc. may be used. Each of the game controller 47, the display device 49, the microphone 51, and the camera 53 is connected to the information processing device 45 so as to be communicable by wire or wirelessly.

[0114] The information processing device 45 is, for example, a stationary game machine. However, it is not limited to this, and for example, a portable game machine integrally provided with an input unit, a display unit, etc. may also be used. In addition to game machines, for example, those manufactured, sold, etc. as computers such as server computers, desktop computers, notebook computers, tablet computers, etc., or those manufactured, sold, etc. as telephones such as smartphones, mobile phones, phablets, etc. may also be used.

[0115] The player performs various operation inputs using the game controller 47. The microphone 51 inputs the voice uttered by the player. The camera 53 detects the orientation of the player's head, the shape of the hand, the movement of the hand, etc. Note that the microphone 51 or the camera 53 may be provided as a single unit as shown in FIG. 12, or may be integrally provided in any of the information processing device 45, the game controller 47, or the display device 49.

[0116] Also, for example, as shown in FIG. 13, a game system 1B (not shown) having a smartphone 55 may be used. The smartphone 55 (an example of an information processing device) is provided with a touch panel 57 on which various displays and various input operations by the player are performed, a voice input function, a camera function capable of detecting hand actions, etc.

[0117] In the above, the case where the voice user interface program of the present invention is a game program has been described as an example, but it is not limited to the game program. For example, when the information processing device is various devices equipped with a voice recognition function such as a car navigation device, an automatic ticket vending machine in a railway, a restaurant, etc., a vending machine, an ATM of a financial institution, an OA device such as a copying machine or a FAX, etc., it may be a voice user interface program applied to those devices.

[0118] In addition to what has been described above, the methods according to the above-described embodiments and each modification example may be appropriately combined and used. Although not exemplified one by one, the above-described embodiments and each modification example may be implemented with various changes made thereto within the scope not departing from the gist thereof.

[0119] <8. Hardware Configuration of Control Unit> Next, with reference to FIG. 14, an example of the hardware configuration of the control unit 7 of the head-mounted display 3 that realizes each processing unit implemented by the program executed by the CPU 301 and the like described above will be described. Note that the information processing apparatus 45 and the smartphone 55 may have a similar hardware configuration.

[0120] As shown in FIG. 14, the control unit 7 includes, for example, a CPU 301, a ROM 303, a RAM 305, a GPU 306, a dedicated integrated circuit 307 constructed for a specific use such as an ASIC or an FPGA, an input device 313, an output device 315, a recording device 317, a drive 319, a connection port 321, and a communication device 323. These components are connected to be able to transmit signals to each other via a bus 309, an input / output interface 311, or the like.

[0121] The game program (an example of an audio user interface program) can be recorded, for example, in a recording device 317 such as a ROM 303, a RAM 305, or a hard disk.

[0122] In addition, the game program can be temporarily or permanently (non-temporarily) recorded on a removable recording medium 325 such as a magnetic disk such as a flexible disk, various optical disks such as CDs, MO disks, and DVDs, or a semiconductor memory. Such a recording medium 325 can also be provided as so-called packaged software. In this case, the game program recorded on these recording media 325 may be read by the drive 319 and recorded in the recording device 317 via the input / output interface 311, the bus 309, or the like.

[0123] Also, the game program can be recorded, for example, on a download site, another computer, another recording device, etc. (not shown). In this case, the game program is transferred via a network NW such as a LAN or the Internet, and the communication device 323 receives this program. Then, the program received by the communication device 323 may be recorded in the recording device 317 via the input / output interface 311, the bus 309, etc.

[0124] Also, the game program can be recorded, for example, in an appropriate external connection device 327. In this case, the game program is transferred via an appropriate connection port 321 and may be recorded in the recording device 317 via the input / output interface 311, the bus 309, etc.

[0125] Then, the CPU 301 executes various processes according to the program recorded in the recording device 317, whereby the processes by the aforementioned voice recognition processing unit 23, first voice determination processing unit 25, first action execution processing unit 27, second voice determination processing unit 29, second action execution processing unit 31, action detection processing unit 33, third action execution processing unit 35, etc. are realized. At this time, the CPU 301 may, for example, directly read and execute the program from the recording device 317, or may execute it after once loading it into the RAM 305. Further, when the CPU 301 receives a program via the communication device 323, the drive 319, or the connection port 321, for example, it may directly execute the received program without recording it in the recording device 317.

[0126] Also, the CPU 301 may perform various processes as necessary based on signals and information input from an input device 313 such as a game controller or, for example, a microphone, a mouse, a keyboard, etc. (not shown).

[0127] The GPU 306 performs processes for image display such as rendering processing, for example, in response to an instruction from the CPU 301.

[0128] Then, the CPU 301 and the GPU 306 output the result of executing the above processing from an output device 315, for example, including the display unit 5 of the aforementioned head-mounted display 3. Further, the CPU 301 and the GPU 306 may transmit this processing result via the communication device 323 or the connection port 321 as necessary, or may record it in the recording device 317 or the recording medium 325.

Explanation of Signs

[0129] 1 Game system 1A Game system 1B Game system 3 Head-mounted display 5 Display unit 7 Control unit (information processing device) 25 First voice determination processing unit 27 First action execution processing unit (first processing execution processing unit) 29 Second voice determination processing unit 31 Second action execution processing unit (second processing execution processing unit) 33 Action detection processing unit 35 Third action execution processing unit (second processing execution processing unit) 45 Information processing device 55 Smartphone (information processing device) 325 Recording medium

Claims

A head-mounted display that can be worn on a player's head, the head-mounted display including a transmissive display unit and an audio input unit that inputs audio emitted by the player, and an information processing device that generates a virtual image based on an input result by the audio input unit and overlays and displays the virtual image on an image in the real space. A first voice determination processing unit that determines whether a first part of a preset word has been uttered by the player based on an input result by the audio input unit. A first processing execution processing unit that, when it is determined that the first part of the word has been uttered, executes a first process related to the virtual image corresponding to the word before the word is uttered to the end. In parallel with the execution of the first process, a second voice determination processing unit that determines whether the player has uttered the word to the end based on an input result by the audio input unit. A second processing execution processing unit that executes a second process related to the virtual image based on a determination result as to whether the word has been uttered to the end. An audio user interface program for functioning as such.

2. The first processing execution processing unit When it is determined that the first part of the word has been uttered, before the word is uttered to the end, as the first process, causes a game character to start a first action corresponding to the word. The second processing execution processing unit When it is determined that the word has not been uttered to the end, as the second process, causes the game character to execute a second action different from the first action. The audio user interface program according to Claim 1.

3. The head-mounted display includes a head direction detection unit that detects the direction of the player's head and a hand movement detection unit that detects the shape or movement of the player's hand, and an information processing device that generates the virtual image based on detection results by the head direction detection unit and the hand movement detection unit and overlays and displays the virtual image on the image in the real space. An action detection processing unit that detects the content of the player's action based on at least one of the direction of the head detected by the head direction detection unit and the shape or movement of the player's hand detected by the hand movement detection unit. To further function as The second processing execution processing unit When it is determined that the word has been uttered to the end, as the second process, based on the content of the first action executed by the game character and the content of the player's action detected, a third action is determined, and the game character is made to execute the third action. The voice user interface program according to claim 2.

4. The first voice determination processing unit determines whether a first part of the word representing a game in which the player and the game character compete for victory or defeat has been uttered by the player. The first process execution processing unit When it is determined that the first part of the word has been uttered, before the word is uttered to the end, as the first process, the game action is started for the game character. The second voice determination processing unit determines whether the player has uttered the word to the end in parallel with the execution of the game action. The second process execution processing unit When it is determined that the word has been uttered to the end, as the second process, based on the content of the game action executed by the game character and the content of the player's action detected, victory or defeat is determined, and the game character is made to execute the third action corresponding to the result of the victory or defeat. The voice user interface program according to claim 3.

5. The second process execution processing unit As the second action, makes the game character execute an action of getting angry with the player. The voice user interface program according to any one of claims 2 to 4.

6. The first voice determination processing unit determines whether a first part of the word "Turn around, ho" has been uttered by the player. The first process execution processing unit When it is determined that the first part of the "Turn around, ho" has been uttered, before the "Turn around, ho" is uttered to the end, the game action of the "Turn around, ho" is started for the game character. The second voice determination processing unit determines whether the player has uttered the word "Turn around, ho" to the end in parallel with the execution of the game action of the "Turn around, ho". The second process execution processing unit When it is determined that the phrase "Turn around, hey" has not been uttered until the end, cause the game character to execute the second action, When it is determined that the phrase "Turn around, hey" has been uttered until the end, determine the winner or loser based on the direction of the face due to the action executed by the game character and the direction of the finger due to the action of the detected player, and cause the game character to execute the third action corresponding to the result of the winner or loser, The voice user interface program according to claim 4.

7. The first voice determination processing unit, Determine whether a first part of the phrase "Jankenpon" has been uttered by the player, The first process execution processing unit, When it is determined that the first part of the "Jankenpon" has been uttered, before the "Jankenpon" is uttered until the end, cause the game character to start the action of the "Jankenpon" game, The second voice determination processing unit, In parallel with the execution of the action of the "Jankenpon" game, determine whether the phrase "Jankenpon" has been uttered until the end by the player, The second process execution processing unit, When it is determined that the phrase "Jankenpon" has not been uttered until the end, cause the game character to execute the second action, When it is determined that the phrase "Jankenpon" has been uttered until the end, determine the winner or loser based on the hand shape due to the action executed by the game character and the hand shape due to the action of the detected player, and cause the game character to execute the third action corresponding to the result of the winner or loser, The voice user interface program according to claim 4.

8. A recording medium readable by an information processing device, which records the voice user interface program according to any one of claims 1 to 7.

9. A head-mounted display that can be worn on the player's head, comprising a transmissive display unit and a voice input unit that inputs the voice emitted by the player, generating a virtual image based on the input result by the voice input unit, and an information processing device that overlays and displays the virtual image on the image in the real space, a game processing method executed by the information processing device, Determining whether a first part of a preset word has been pronounced by the player based on an input result from the voice input unit; When it is determined that the first part of the word has been pronounced, executing a first process related to the virtual image corresponding to the word before the word is pronounced to the end; In parallel with the execution of the first process, determining whether the word has been pronounced to the end by the player based on an input result from the voice input unit; Executing a second process related to the virtual image based on a determination result of the step of determining whether the word has been pronounced to the end; A voice user interface processing method comprising the above steps.

Images