Program, information processing method, information processing device, and information processing system

Abstract

The present application can improve the usability when a character is interactively controlled. A program in the present application causes an information processing apparatus to perform the following processing: acquire at least an electrooculogram signal from another device worn on the head of a user, detect at least movement of the user's eye based on the electrooculogram signal, superimpose a character onto an image being displayed in a screen and perform display control, and control the action of the character based on the detection result of the movement of the eye.

Description

Technical Field

[0001] This invention relates to a program, an information processing method, an information processing apparatus, and an information processing system. Background Technology

[0002] Previously, there was a known technology that controlled an avatar overlaid as augmented reality based on head movements detected by a head-mounted display (HMD) worn by the user (see, for example, Patent Documents 1 and 2).

[0003] Patent Document 1: Japanese Patent No. 6470859

[0004] Patent Document 2: Japanese Patent Application Publication No. 2018-069069 Summary of the Invention

[0005] However, in the existing technology, the avatar (character) is controlled by using head movement detected by HMD, and the camera is used to take pictures by using the movement of the face. Therefore, it is not easy to overlay the avatar onto the image, which may not improve the usability for interactive character control.

[0006] Therefore, the purpose of this invention is to improve the usability of interactive control of characters.

[0007] In one embodiment of the present invention, the program performs the following processing in an information processing device: acquiring an eye potential signal from at least another device worn on the user's head, detecting the movement of the user's eyes based at least on the eye potential signal, overlaying a character onto an image being displayed on the screen and controlling its display, and controlling the character's actions based on the detection result of the eye movement.

[0008] According to the present invention, usability can be improved when interactively controlling a character. Attached Figure Description

[0009] Figure 1 This is a diagram illustrating an example of the information processing system in the embodiment.

[0010] Figure 2 This is a schematic structural diagram illustrating the hardware structure of the information processing device in the embodiment.

[0011] Figure 3 This is a block diagram illustrating an example of the structure of the processing apparatus in the embodiment.

[0012] Figure 4 This is a diagram illustrating an example of the structure of the information processing apparatus in the embodiment.

[0013] Figure 5 This is a diagram of Example 1 showing the display screen of application A in the embodiment.

[0014] Figure 6 This is a diagram of Example 2 showing the display screen of Application A in the embodiment.

[0015] Figure 7 The figure is shown in Example 3, illustrating the display screen of application A in the embodiment.

[0016] Figure 8 The figure is shown in Example 4, illustrating the display screen of application A in the embodiment.

[0017] Figure 9 The figure in Example 5 shows the display screen of application A in the embodiment.

[0018] Figure 10 The figure is shown in Example 6, illustrating the display screen of application A in the embodiment.

[0019] Figure 11 This is a timing diagram illustrating an example of the processing of application A in the embodiments. Detailed Implementation

[0020] The embodiments of the present invention will now be described with reference to the accompanying drawings. However, the embodiments described below are merely examples and are not intended to exclude the application of various modifications and techniques not explicitly shown below. That is, the present invention can be modified in many ways without departing from its spirit. Furthermore, in the accompanying drawings below, the same or similar parts are indicated by the same or similar symbols. The drawings are schematic diagrams and do not necessarily correspond to actual dimensions, proportions, etc. Even between different drawings, there may be parts that differ in size and proportion.

[0021] [Example]

[0022] In the embodiments, although eyewear is cited as an example of a wearable terminal equipped with an accelerometer, an angular velocity sensor, and bioelectrodes, it is not limited to this. Figure 1 This is a diagram illustrating an example of information processing system 1 in the embodiment. Figure 1 The information processing system 1 shown includes an information processing device 10 and an eye-wearing device 30, which are connected via a network to enable data communication.

[0023] The eye-worn device 30, for example, has a processing device 20 mounted on its bridge portion. The processing device 20 includes a pair of nose pads and a bridge portion, each having bioelectrodes 32, 34, and 36. The processing device 20 may also include a triaxial accelerometer and a triaxial angular velocity sensor (or a six-axis sensor).

[0024] The processing device 20 detects sensor signals, eye potential signals, etc., and sends them to the information processing device 10. The processing device 20 is not necessarily located on the central beam; it can be positioned where it can acquire eye potential signals when the eye-wearing device 30 is worn. Furthermore, the processing device 20 can also be detachable from the central beam.

[0025] The information processing device 10 is an information processing device with communication functions. For example, the information processing device 10 is preferably a mobile terminal such as a smartphone held by the user, but it can also be a personal computer, a tablet terminal, etc. The information processing device 10 detects the user's eye movement and head movement based on eye potential signals or sensor signals received from the processing device 20, and controls the character (e.g., avatar) superimposed on the image being displayed on the screen based on the detection results.

[0026] <Hardware Structure of Information Processing Device 10>

[0027] Figure 2 This is a schematic structural diagram illustrating the hardware structure of the information processing device 10 in the embodiment. A typical example of the information processing device 10 is a mobile terminal such as a smartphone, but in addition to this, mobile terminals that can be wirelessly or wiredly connected to a network, electronic devices equipped with a touch panel such as tablet terminals, and general-purpose devices that can process data and display images while communicating using a network are also included in the information processing device 10 in the embodiment.

[0028] The information processing device 10 in this embodiment includes, for example, a rectangular thin housing (not shown), on one side of which a touchscreen 102 is formed. In the information processing device 10, each structural unit is connected to a main control unit 150. The main control unit 150 is, for example, one or more processors.

[0029] The main control unit 150 is connected to a mobile communication antenna 112, a mobile communication unit 114, a wireless LAN (Local Area Network) communication antenna 116, a wireless LAN communication unit 118, a storage unit 120, a speaker 104, a microphone 106, a hard button 108, a hard key 110, and a six-axis sensor 111. Furthermore, the main control unit 150 is further connected to a touchscreen 102, a camera 130, and an external interface 140. The external interface 140 includes an audio output terminal 142.

[0030] The touchscreen 102 functions as both a display device and an input device, and consists of a display (display screen) 102A for display and a touch sensor 102B for input. The display 102A is a common display device, such as a liquid crystal display or an organic EL (electroluminescence) display. The touch sensor 102B is configured to have an element for detecting touch operations disposed on the upper surface of the display 102A, and a transparent operating surface superimposed thereon. The touch sensor 102B can employ any known method for contact detection, such as capacitive, resistive film (pressure-sensitive), or electromagnetic induction.

[0031] The touchscreen 102, serving as a display device, displays an image of an application, which is generated through the execution of program 122 based on the main control unit 150. The touchscreen 102, serving as an input device, receives operation input by detecting the movement of an object touching the operating surface (including a user's finger or stylus, which will be described below as a "finger" example), and provides information about the contact position to the main control unit 150. The detected finger movement is used as coordinate information representing the position or area of ​​the contact point; this coordinate information is, for example, represented as coordinate values ​​on two axes of the touchscreen 102, along its short and long sides.

[0032] The storage unit 120 stores a program 122 that performs processing related to the character superimposed on the image being displayed. The storage unit 120 may also be provided separately from the information processing device 10, for example, it may be a storage medium such as an SD card (Secure Digital Memory Card) or CD-ROM (CD-Random Access Memory) that can be read by a computer, or a non-temporary storage medium.

[0033] The information processing device 10 is connected to the network N via a mobile communication antenna 112 or a wireless LAN communication antenna 116, enabling data communication between the information processing device 10 and the processing device 20.

[0034] <Structure of Processing Unit 20>

[0035] Figure 3 This is a block diagram illustrating an example of the structure of the processing apparatus 20 in the embodiment. For example... Figure 3 As shown, the processing device 20 includes a processing unit 202, a transmitting unit 204, a six-axis sensor 206, a power supply unit 208, and bioelectrodes 32, 34, and 36. Furthermore, each bioelectrode 32, 34, and 36 is connected to the processing unit 202, for example, via an amplification unit and using wires.

[0036] The six-axis sensor 206 is a three-axis accelerometer and a three-axis angular velocity sensor. Alternatively, these sensors can be installed separately. The six-axis sensor 206 outputs the detected sensor signals to the processing unit 202.

[0037] The processing unit 202 processes the sensor signals obtained from the six-axis sensor 206 or the eye potential signals obtained from the various bioelectrodes 32, 34, and 36 as needed. For example, it digitizes the sensor signals and eye potential signals into data packets and outputs the data packets to the transmitting unit 204. Furthermore, the processing unit 202 includes a processor, which can use the eye potential signals to calculate first bio-information about blinking and second bio-information about gaze movement.

[0038] Furthermore, the processing unit 202 can also use the sensor signals from the six-axis sensor 206 to calculate third biometric information about head movement. This information about head movement includes, for example, information about the head moving forward, backward, left, and right. Additionally, the processing unit 202 can also amplify only the sensor signals acquired from the six-axis sensor. Hereinafter, the processing unit 202 will be described as a structure that performs data packet processing of the ocular potential signals and sensor signals.

[0039] The transmitting unit 204 transmits the iris potential signal and / or sensor signal, which have been data-packed by the processing unit 202, to the information processing device 10. For example, the transmitting unit 204 transmits the iris potential signal and sensor signal to the information processing device 10 via wireless or wired communication such as Bluetooth (registered trademark) and wireless LAN. The power supply unit 208 supplies power to the processing unit 202, the transmitting unit 204, and the six-axis sensor 206.

[0040] <Structure of Information Processing Device 10>

[0041] Next, the structure of the information processing device 10 will be described. Figure 4 This figure illustrates an example of the structure of the information processing apparatus 10 in the embodiment. The information processing apparatus 10 includes a storage unit 302, a communication unit 304, and a control unit 306.

[0042] Storage unit 302 can, for example Figure 2 This is implemented using storage units such as 120. As an example, storage unit 302 stores data about an application (hereinafter also referred to as application A) that performs processing to generate an image overlaid with a character using AR (Augmented Reality) technology. Data about application A includes, for example, data received from processing device 20, information about the display and control of the character, image data of the overlaid character, and image information displayed on the screen. The character may include, for example, an avatar, and the image may include a still image or a moving image.

[0043] The communication unit 304 can be implemented, for example, by a mobile communication unit 114 or a wireless LAN communication unit 118. The communication unit 304 receives data from the processing device 20, for example. Furthermore, the communication unit 304 can also send data processed in the information processing device 10 to a server. That is, the communication unit 304 functions as both a transmitter and a receiver.

[0044] The control unit 306 can be implemented, for example, by a main control unit 150. The control unit 306 executes application A. In this embodiment, application A acquires eye potential signals and / or sensor signals, and detects the user's eye movement and head movement based on these signals, then controls the character's actions based on the detection results. Furthermore, the control unit 306 overlays the controlled character onto an image, generates an image containing the character, and saves it. To achieve the above functions, the control unit 306 includes an acquisition unit 312, a detection unit 314, a display control unit 316, a character control unit 318, and an operation detection unit 320.

[0045] The acquisition unit 312 acquires the signals received by the communication unit 304. For example, the acquisition unit 312 acquires at least the endoscopic potential signal from other devices worn on the user's head (e.g., eye-wearing device 30).

[0046] The detection unit 314 detects at least the user's eye movements based on the ocular potential signal acquired by the acquisition unit 312. For example, the detection unit 314 detects eye movements, including blinking or gaze movement, based on the ocular potential signal using known techniques.

[0047] The display control unit 316 overlays the character onto the image being displayed on the screen (display 102A) of the information processing device 10 and controls its display. The image can be an image selected by the user or an image being captured by the camera 130.

[0048] The character control unit 318 controls the character's movements based on the detection results of eye movements detected by the detection unit 314 and / or instructions described later. For example, the character control unit 318 controls the blinking and gaze movements of the character superimposed on the image in sync with the detected blinking and gaze movements. The character can be, for example, a pre-set avatar or an avatar selected by the user.

[0049] The character control unit 318 has multiple motion parameters for controlling the character's movement, allowing the user's blinking, gaze movement, and head movement to be associated with each of the character's motion parameters. This association can also be performed based on user input. The multiple motion parameters include, for example, parameters related to the character's blinking, gaze movement, head movement, limb movement, shrinking / enlarging of the character, and hand movement.

[0050] The operation detection unit 320 detects user operations on UI (User Interface) elements displayed on the screen and outputs various commands to the corresponding parts based on the user operations. For example, if the operation detection unit 320 detects operations such as the character selection button, the character's basic action button, or the character's expression button, it outputs the command corresponding to the detected operation to the character control unit 318.

[0051] Based on the above, usability for interactive control of a character can be improved by using eye potential signals obtained from the user who is instructing the character, without using a camera to photograph the face. For example, various controls can be performed based on eye potential signals, and more precise control can be achieved over movements such as the character's eyes.

[0052] Alternatively, the acquisition unit 312 may also include acquiring sensor signals sensed by an accelerometer and / or angular velocity sensor mounted on other devices (e.g., the eye-wearing device 30). In this case, the detection unit 314 may also include detecting the movement of the user's head based on the sensor signals. Furthermore, the character control unit 318 may also control the character's movements based on the detection results of the user's eye movements and head movements.

[0053] Regarding the character's actions, this includes, for example, controlling the character's blinking and gaze movements to synchronize with the user's blinking and gaze movements, or controlling the character's head movements to synchronize with the user's head movements. Furthermore, the character's actions may also include, for example, determining the action parameters for a specified action A based on the user's blinking and gaze movements, or determining the action parameters for a specified action B based on the user's head movements.

[0054] Therefore, usability for interactive control of a character can be improved by incorporating head movements of the user who is instructing the character. For example, by using eye potential signals and head movements, it is possible to increase the variety of controlled objects.

[0055] The display control unit 316 may also display and control a UI component on the screen of the information processing device 10, allowing selection of basic movements related to the character's limbs. In this case, the character control unit 318 may also include controlling the character's movements based on basic movements related to the character's limbs and movements related to the character's head, wherein the basic movements related to the character's limbs are selected by the user using UI components, and the movements related to the character's head are based on detection results.

[0056] For example, the display control unit 316 controls the display of a selection button on the screen, which allows the user to select a pre-set basic movement related to the character's limbs. Furthermore, the character control unit 318 controls the display so that the basic movements based on the instructions corresponding to the buttons selected by the operation detection unit 320 are reflected in the basic movements of the displayed character's limbs.

[0057] Therefore, character movements are divided into head movements and limb movements. Head movements can be controlled based on the user's eye and head movements, while limb movements can be easily controlled via UI components on the screen. This improves the usability of interactive character control.

[0058] As described above, by using an eye-worn device 30 equipped with a biometric measurement system, the functions of head-related operations and limb-related operations of the avatar can be separated. Therefore, by displaying the character's body operation interface on the screen, even complex and multifaceted operations can be performed intuitively without the need for expensive external wearable terminals.

[0059] Furthermore, the display control unit 316 can also overlay a character onto the image being captured by the photographic device (e.g., camera 130) and perform display control. For example, when the user activates the camera 130, the display control unit 316 can use AR technology to overlay a character onto the image being captured, the character control unit 318 can control the actions of the overlaid character, and the control unit 306 can save the image containing the overlaid character.

[0060] Therefore, there is no need to use camera 130 to capture the user's eye movements, etc. Thus, camera 130 can capture images of superimposed characters and easily achieve real-time image capture, character superimposition, and control. For example, it is possible to use the rear camera of a mobile terminal to capture animation while interactively controlling the superimposed character.

[0061] Furthermore, the detection unit 314 may also include the detection of blinking or gaze movement based on ocular potential signals. Known methods can be used for blinking or gaze movement detection. In this case, the character control unit 318 may include control of the character's movements based on a first action parameter associated with blinking or a second action parameter associated with gaze movement. For example, the character control unit 318 may control the user's blinking or gaze movement to be reflected in the character's blinking (first action parameter) or gaze movement (second action parameter) in real time. Furthermore, the character control unit 318 may also control the user's blinking or gaze movement in correspondence with the action parameters of two other characters, for example, in correspondence with the first or second action parameters relating to the character's limb movements.

[0062] Therefore, detection results obtained from ocular potential signals such as blinking and gaze movement can be used to improve the usability of interactive control of characters. Furthermore, it allows for the addition of variations in the controlled object.

[0063] Additionally, the detection unit 314 may include detection of the intensity of blinking or the speed of gaze movement based on ocular potential signals. For example, the detection unit 314 may set multiple thresholds for signal intensity based on ocular potential signals and detect the intensity of blinking by dividing it into multiple levels. Similarly, for the speed of gaze movement, it may set multiple thresholds for horizontal movement speed and detect the speed of gaze movement by dividing it into multiple levels.

[0064] In this case, the character control unit 318 can also control the character's movements based on a third motion parameter associated with the intensity of the blink or a fourth motion parameter associated with the speed of gaze movement. The user can preset which motion parameter the intensity of the blink or the speed of gaze movement is associated with. For example, the character control unit 318 can change the magnitude of the character's movement based on the intensity of the blink; for instance, the stronger the blink, the greater the amplitude of the character's wave. Furthermore, the character control unit 318 can change the speed of the character's movements based on the speed of gaze movement; for instance, the faster the gaze movement, the faster the character's swaying cycle.

[0065] Therefore, it is possible to improve the usability of interactive control of characters by using parameters unique to ocular potential signals, such as the intensity of blinking and the speed of gaze movement. Furthermore, it is possible to increase the variability of the controlled object.

[0066] Additionally, the character control unit 318 may also include the ability to change the position of the character relative to the depth direction of the screen based on head movement detected in the detection results. For example, the character control unit 318 may perform the following control: a virtual camera is positioned at a designated location in front of the screen of the information processing device 10 (on the user's side of the screen); if the user's head tilts forward, the character is moved closer to the virtual camera, and the size of the character on the screen increases; if the user's head tilts backward, the character is moved away from the virtual camera, and the size of the character on the screen decreases.

[0067] It should be noted that the controlled part of the character can be the entire displayed character, the limbs and above, or the head. It can be predetermined which part to control, or the controlled part can be increased based on the user's head tilt. For example, as the user's head tilts more, the controlled part can be increased in the order of head, limbs and above, and then the entire displayed character.

[0068] Therefore, the user's head movements can be used to control the distance and size of the character, thereby improving the usability of interactive character control.

[0069] Furthermore, a character's actions can include active and passive actions. Active actions are actions performed automatically by the character, such as repeatedly executing a specified action within a specified period after receiving a designated instruction or command. Passive actions are actions performed each time an instruction is received. Active actions include, for example, repeating a fixed movement like swaying left and right if the user sets an active action. Passive actions include, for example, performing a designated action such as greeting or showing surprise after receiving a designated instruction or command, and not performing the action until the next instruction or command is received.

[0070] In this case, the character control unit 318 may also include parameters that determine the value of active actions based on the detection results. For example, if the active action is a swaying movement, the character control unit 318 may determine parameters related to the period of the swaying movement based on the number of blinks within a specified period.

[0071] Therefore, even when the character performs actions automatically, the user's unconscious actions can be reflected in the character by using the user's eye movements or head movements, thereby improving the usability of interactive control of the character.

[0072] Furthermore, the detection unit 314 may also include a known technique for detecting the user's level of concentration or calmness based on blinks detected from eye potential signals. An example of a known technique is the one described in the applicant's Japanese Patent Application Publication No. 2017-70602, which describes the detection of concentration and calmness using blinks and eye movements.

[0073] Furthermore, the character control unit 318 can also control the character's movements based on the level of concentration or calmness contained in the detection results. For example, as an example of character movement control, the character control unit 318 can change the character's expression to a focused expression when it determines that the user is focused, and change the character's expression to a relaxed expression when it determines that the user is calm, and change the character's facial expression, the brightness and color saturation of the image containing the character according to the user's concentration or calmness.

[0074] Therefore, it is possible to utilize psychological states that users are not consciously aware of (such as concentration or calmness) and reflect these psychological states in the character, thereby improving the smoothness of the character's actions when interacting with the character.

[0075] Furthermore, the information processing device 10 can be a mobile terminal such as a smartphone, which is an example of a mobile processing terminal, and other devices can be an eye-wearing device 30, which is an example of a wearable terminal.

[0076] Therefore, since users generally own mobile devices, portability and versatility will be improved, and movement can be performed while sensing eye potential signals, thereby increasing the dynamic changes of the superimposed character's image. For example, it is possible to use the mobile device's external camera to shoot animations while interactively controlling the superimposed character.

[0077] Furthermore, the character control unit 318 can also change the character to another character using eye movement. For example, the character control unit 318 sets a threshold for the amount of eye movement, and changes the character displayed on the screen when the movement exceeds the threshold. It should be noted that the character control unit 318 can also begin processing the character change via eye movement if the user presses the character change button beforehand. In this way, the character control unit 318 can also associate the user's eye movement with various operations related to the character.

[0078] Therefore, by associating the user's eye movements with various actions related to the character, usability when controlling the character can be improved.

[0079] <Screen example>

[0080] Next, use Figures 5-10 An example of a screen display for application A in the embodiment will be described. Figure 5 This is a diagram illustrating Example 1, showing the display screen of application A in the embodiment. Figure 5 In the screen D10 shown, for example, a selection button B10 for facial behaviors (e.g., 5 types) is displayed on the left, and a selection wheel (revolver) B12 for body behaviors (e.g., 27 types) is displayed on the right. The operation detection unit 320 detects the operation in this screen, thereby the character control unit 318 determines the basic facial expression (basic movement) of the character C10 and determines the basic limb movements of the character C10.

[0081] Figure 6 This is a diagram of Example 2 showing the display screen of Application A in the embodiment. Figure 6 The screen D20 shown illustrates an example of erasing the character C20 in the upper left corner using the wipe button. Even after erasing, facial and body actions can still be performed via the character control unit 318.

[0082] Figure 7 The figure is shown in Example 3, illustrating the display screen of application A in the embodiment. Figure 7 Screen D30 shows an example of a screen displaying a button area in the upper right corner, and the screen when the gear button B30 is clicked from that button area. Clicking the gear button B30 allows you to select an overlay character image.

[0083] For example, to import still images, the user clicks "Import photo," and to import videos, they click "Import video." Furthermore, to use the rear camera to capture videos, the user clicks "Use camera." It should be noted that a white background can be included in the initial settings for the selected object. Additionally, imported images and videos will automatically be added to the D30 of the screen, allowing the user to select the desired image by clicking.

[0084] Figure 8 The figure is shown in Example 4, illustrating the display screen of application A in the embodiment. Figure 8 Screen D40 shows an example of a screen displaying a button area in the upper right corner, and how clicking the humanoid button B40 occurs. In this example, clicking the humanoid button B40 allows selection of basic body movements for the character. For example, 27 basic body movements can be prepared. For instance, if the user selects 12 from the 27, the 12 basic movements are increased to [number missing]. Figure 5 The selection wheel B12 is shown.

[0085] It should be noted that, as mentioned above, basic actions (behaviors) can include both "active actions" and "passive actions." "Active actions" include "swaying," "walking," "running," "specifying a sitting posture," "playing games," "using an air chair," "eating," and "standing up," etc. "Passive actions" basically include actions other than "active actions."

[0086] Figure 9 The figure in Example 5 shows the display screen of application A in the embodiment. Figure 9 Screen D50 shows an example of a screen displaying a button area in the upper right corner, and how clicking the model button B50 from that area will select a character model. In this example, clicking the model button B50 will allow you to select a character model.

[0087] The character models include default models pre-set in application A, as well as models created using VRM files for 3D avatars. Users can import specified VRM files into application A beforehand. When importing a VRM file, users can click "Import VRM" and use it.

[0088] Figure 10 The figure is shown in Example 6, illustrating the display screen of application A in the embodiment. Figure 10 Screen D60 shows an example of a screen displaying a button area in the upper right corner, and the user clicking the file button B60 from that area. In this example, clicking the file button B60 allows the user to select and view pre-made character image data overlaid on it. Captured images, including moving images and still images, can be simultaneously saved in the image application of the information processing device 10. It should be noted that the image file overlaid with the character is output in MP4 file format, making it easier to export, share to SNS (Social Networking Services), and perform other processing tasks.

[0089] <Action>

[0090] Next, the various processes of the information processing system 1 in the embodiment will be described. Figure 11 This is a timing diagram illustrating an example of the processing of application A in the embodiments.

[0091] exist Figure 11 In step S102 shown, the control unit 306 starts application A according to the user's operation.

[0092] In step S104, the control unit 306 establishes communication with the eye-wearing device 30. The communication method is, for example, Bluetooth (registered trademark), Wi-Fi (registered trademark), etc.

[0093] In step S106, the processing device 20 of the eye-wearing device 30 uses bioelectrodes 32, 34, and 36 to measure the user's ocular potential. Furthermore, if the processing device 20 includes an accelerometer / angular velocity sensor, measurements based on these sensors are also performed.

[0094] In step S108, the processing device 20 of the eye-wearing device 30 sends the acquired ocular potential signal and / or sensor signal to the information processing device 10.

[0095] In step S110, the acquisition unit 312 of the information processing device 10 acquires the ocular potential signal transmitted from the eye-wearing device 30 worn on the user's head. Furthermore, the acquisition unit 312 may also acquire sensor signals indicating the movement of the user's head.

[0096] In step S112, the detection unit 314 of the information processing device 10 detects the user's eye movement based on the acquired ocular potential signal. In addition, the detection unit 314 can also detect blinking and gaze movement as eye movement, and when sensor signals are acquired, it can also detect head movement (front-back direction of the face, horizontal direction of the face, vertical direction of the face, etc.) and include it in the detection result.

[0097] In step S114, the display control unit 316 overlays the character onto the image currently displayed on the screen and controls its display. It should be noted that the displayed image can also be selected by the user.

[0098] In step S116, the character control unit 318 controls the movement of the character superimposed on the image and displayed based on the detection results of the user's eye movement. Furthermore, if the detection results include the user's head movement, the character control unit 318 can also control the character's movement based on that head movement.

[0099] In step S118, the control unit 306 stores image data containing the superimposed character's movements in the storage unit 120 of the information processing device 10. For example, the image data is not limited to being saved in MP4 format; it can be saved in a file format corresponding to the target.

[0100] In step S120, in order to upload to SNS or attach to email, the control unit 306 outputs the saved image data containing the overlaid character to an external device, etc.

[0101] It should be noted that for those included in Figure 11The processing steps described in the document can be ordered or executed in parallel without causing contradictions in the processing content. Additionally, additional steps can be added between processing steps. Furthermore, a step that is recorded as a single step for convenience can be divided into multiple steps for execution, while a step that is recorded as multiple steps for convenience can be treated as a single step.

[0102] As described above, the usability of interactive control of characters can be improved according to the embodiments. Furthermore, in the case of controlling 3D models, according to the embodiments, by inputting a recorded VRM data model onto a humanoid, it is possible to overlay the user-desired character (e.g., avatar) onto the image.

[0103] Furthermore, users can control the character superimposed on the image by moving their eyes and head. Additionally, according to an embodiment, users can manipulate the character's facial and body movements in a manner similar to a game controller (see, for example, see...). Figure 5-10 ).

[0104] Furthermore, regarding the overlaid image, it is possible to capture AR-synthesized dynamic images and still images using the rear camera of a mobile terminal or similar device. Additionally, it is possible to use pre-captured dynamic or still images as a background to synthesize AR characters (e.g., avatars) and interact with them.

[0105] Furthermore, based on the aforementioned role control, by transferring a portion of the control to an eye-wearing device 30 equipped with a bio-information measurement system, intuitive operation without using a camera can be performed, enabling portable movement of the sensing and image-capturing device.

[0106] Furthermore, by using an eye-worn device 30 equipped with a bio-information measurement system as an application A of the information processing device 10, AR compositing does not require editing capabilities and can be performed using an easily editable UI.

[0107] Furthermore, as described above, it is possible to use the external camera of the information processing device 10 to capture images, to capture images while performing AR synthesis, and to control intuitively controllable characters.

[0108] Furthermore, it enables interactive shooting of characters using AR from a third-person perspective, rather than the previous AR-based viewing character shooting applications.

[0109] It should be noted that in this embodiment, the case where the eye-wearing device 30 is eyeglasses has been described. However, the eye-wearing device is not limited to this. The eye-wearing device can be any device associated with the eyes, such as eyeglasses, sunglasses, goggles, head-mounted displays, and the supports for these devices, etc., for face or head use.

[0110] It should be noted that, in this embodiment, sensor signals from a six-axis sensor mounted on the eye-wearing device 30 are used for explanation, but the applications described in the embodiment can be performed even using sensor signals from a six-axis sensor 111 mounted on the information processing device 10. That is, the six-axis sensor can be worn not only on the head but also on any part of the body.

[0111] While the present invention has been described above using embodiments, its technical scope is not limited to the scope described in the above embodiments. Those skilled in the art will understand that various changes or improvements can be made to the above embodiments. These changes or improvements can also be included within the technical scope of the present invention, as can be clearly seen from the claims.

[0112] Symbol Explanation

[0113] 10. Information processing device

[0114] 20 processing devices

[0115] 30 Eyewear

[0116] 302 Storage Department

[0117] 304 Ministry of Communications

[0118] 306 Control Department

[0119] 312 Acquisition Department

[0120] 314 Detection Department

[0121] 316 Display Control Unit

[0122] 318 Character Control Department

[0123] 320 Operation and Inspection Department.

Claims

1. An information processing method, wherein, The information processing device performs the following steps: The acquisition step includes acquiring at least an ocular potential signal from another device worn on the user's head, and includes acquiring sensor signals sensed by an accelerometer and / or angular velocity sensor mounted on the other device. The detection step includes detecting at least the movement of the user's eyes based on the eye potential signal, and also includes detecting the movement of the user's head based on the sensor signal. The display control steps involve overlaying the character onto the image currently being displayed on the screen and displaying UI components on the screen that allow selection of basic actions related to the character's limbs. as well as The motion control steps are as follows: the motion control steps are to control the character's motion based on basic limb-related motions selected using the UI components and head-related motions based on the detection results of eye movement, and include changing the character's position relative to the depth direction of the screen according to the head movement.

2. The information processing method as described in claim 1, wherein, The display control step includes overlaying the character onto the image being captured by the camera device and performing display control.

3. The information processing method as described in claim 1, wherein, The detection steps include detecting blinking or gaze movement based on the ocular potential signal. The motion control steps include controlling the character's actions based on a first motion parameter associated with the blink or a second motion parameter associated with the gaze movement.

4. The information processing method as described in claim 1, wherein, The detection step includes detecting the intensity of blinking or the speed of gaze movement based on the ocular potential signal. The motion control steps include controlling the character's movements based on a third motion parameter associated with the intensity of the blink or a fourth motion parameter associated with the speed of the gaze movement.

5. The information processing method as described in claim 1, wherein, The character's actions include active actions performed automatically. The motion control step includes determining the values ​​of parameters related to the active motion based on the detection results.

6. The information processing method as described in claim 1, wherein, The detection steps include using blinks detected based on the ocular potential signal to detect the user's level of concentration or calmness. The motion control steps include controlling the character's actions based on the level of concentration or the level of calm contained in the detection results.

7. The information processing method as described in claim 1, wherein, The information processing device is a mobile terminal, and the other devices are eye-wearing devices.

8. A computer readable storage medium, wherein, The storage medium stores a program that causes a computer to perform the following steps: The acquisition step includes acquiring at least an ocular potential signal from another device worn on the user's head, and includes acquiring sensor signals sensed by an accelerometer and / or angular velocity sensor mounted on the other device. The detection step includes detecting at least the movement of the user's eyes based on the eye potential signal, and also includes detecting the movement of the user's head based on the sensor signal. The display control steps involve overlaying the character onto the image currently being displayed on the screen and displaying UI components on the screen that allow selection of basic actions related to the character's limbs. as well as The motion control steps are as follows: the motion control steps are to control the character's movements based on basic movements related to the character's limbs selected using the UI components and movements related to the character's head based on the detection results of eye movement, and include changing the position of the character relative to the depth direction of the screen according to the movement of the head.

9. An information processing apparatus comprising one or more processors, wherein, The one or more processors perform the following steps: The acquisition step includes acquiring at least an ocular potential signal from another device worn on the user's head, and includes acquiring sensor signals sensed by an accelerometer and / or angular velocity sensor mounted on the other device. The detection step includes detecting at least the movement of the user's eyes based on the eye potential signal, and also includes detecting the movement of the user's head based on the sensor signal. The display control steps involve overlaying the character onto the image currently being displayed on the screen and displaying UI components on the screen that allow selection of basic actions related to the character's limbs. as well as The motion control steps are as follows: the motion control steps are to control the character's movements based on basic movements related to the character's limbs selected using the UI components and movements related to the character's head based on the detection results of eye movement, and include changing the position of the character relative to the depth direction of the screen according to the movement of the head.

10. An information processing system, the information processing system comprising an eye-wearing device and an information processing apparatus capable of data communication connection, wherein, The eye-wearing device includes: Multiple bioelectrodes; and The transmitting unit transmits the eye potential signals acquired from the plurality of bioelectrodes to the information processing device. The information processing device includes: The communication unit receives at least the ocular potential signal from the eye-wearing device, and receives sensor signals sensed by the accelerometer and / or angular velocity sensor mounted on the eye-wearing device. as well as The control unit detects at least the user's eye movement based on the eye potential signal, detects the user's head movement based on the sensor signal, overlays a character onto the image being displayed on the screen, and displays a UI component that allows selection of basic movements related to the character's limbs on the screen for control. It controls the character's movements based on the basic movements related to the character's limbs selected using the UI component and the movements related to the character's head based on the detection results of the eye movement. It also changes the character's position relative to the depth direction of the screen based on the head movement.

Images