VIRTUAL SPACE MOVING DEVICE, VIRTUAL SPACE MOVING METHOD, AND VIRTUAL SPACE MOVING PROGRAM

Abstract

To enable a user to persistently exercise in an enjoyable manner without getting bored.SOLUTION: A virtual space movement device comprises: a storage unit that stores movement course information including a landscape where a character in a virtual space moves; an acquisition unit that acquires biological information indicating a biological state of a user who exercises and exercise amount information indicating the exercise amount of the user; a calculation unit that calculates movement quality information indicating the quality of the user's exercise and the quality of the movement of the character interlocked with the user based on the biological information and the exercise amount information; and a control unit that controls the movement course information by changing the view point in the virtual space based on the movement quality information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a virtual space movement device, a virtual space movement method, and a virtual space movement program. [Background technology]

[0002] In recent years, the fat burning effect of aerobic exercise is said to be very effective not only for dieting but also for improving health. This is because the oxygen absorbed into the blood by aerobic exercise promotes fat burning. In addition, continuous exercise increases muscle mass and increases basal metabolic rate, creating a virtuous cycle that burns more fat. There are various methods of aerobic exercise, such as jogging and swimming, but walking is particularly popular because it is easy to do. However, elderly people tend to have a decreased ability to detect danger, including a decline in eyesight, so there is an increasing need for technology that can support their movement. From the perspective of efficiently burning fat through aerobic exercise, it is most important to absorb oxygen into the blood, and care must be taken not to exercise anaerobicly by going too fast.

[0003] Therefore, it is necessary to know the load caused by exercise first. Since the load caused by exercise varies from person to person depending on physical strength, etc., it is effective to use the heart rate (pulse rate), which is an individual's biological information, as a guide, rather than the objective amount of exercise. This is why many athletes train while watching their heart rate. The heart rate suitable for burning fat is generally said to be around 120, about 70% of the maximum heart rate, although this depends on age, etc. For this reason, various mobile devices that can measure biological information such as heart rate have been proposed.

[0004] For example, Patent Document 1 describes a technology relating to an information and communication terminal such as a mobile phone capable of measuring biological information such as heart rate, and a biological information system. With this technology, biological information that cannot be measured by the information and communication terminal can also be stored and managed in the information and communication terminal.

[0005] In addition, Patent Document 2 describes a portable vital sign monitor, and in particular a portable vital sign monitor using a blood flow meter that measures information about the blood flow in a target vital sign by using scattered light from the vital sign. A wristwatch with a heart rate monitor function has also been proposed.

[0006] For example, Patent Document 3 describes a technology related to a wristwatch-type exercise measuring device that can measure exercise intensity and heart rate while being worn on the arm. This technology measures the heart rate based on the cardiac potential from a pair of electrodes, and the same principle can be applied to mobile terminals. A mobility support device equipped with a heart rate monitor has also been proposed.

[0007] For example, Patent Document 4 describes a portable navigation system, particularly a technology that provides information that can contribute to health promotion. This technology can display the calorie consumption calculated from the heart rate for each road section in association with map data, thereby supporting outdoor fitness activities in a more enjoyable and effective manner. However, this technology only displays "which parts of the map contributed to calorie consumption and which parts were difficult," and is intended to retrospectively evaluate calorie consumption through exercise. Meanwhile, a mobile terminal device using augmented reality technology has also been proposed.

[0008] For example, Patent Document 5 describes a technique relating to an annotation display system and method that can change the amount of annotation information displayed on a mobile terminal in accordance with the moving speed of the mobile terminal.

[0009] That is, the virtual space movement device according to the first aspect has a storage unit that stores movement course information including scenery when a character moves in the virtual space, , Yu Shows the momentum of the user Exercise information An acquisition unit for acquiring Exercise information Based on We evaluated the user's motion and the movement of the character that moves in the virtual space in conjunction with the user. The device includes a calculation unit that calculates movement quality information, and a control unit that controls movement course information by changing a viewpoint in the virtual space based on the movement quality information. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] JP 2014-120006 A [Patent Document 2] Patent No. 4119862 [Patent Document 3] Patent No. 4685705 [Patent Document 4] Patent No. 4822208 [Patent Document 5] JP 2007-292505 A [Patent Document 6] JP 2011-123807 A [Patent Document 7] Patent No. 4767816 Summary of the Invention [Problem to be solved by the invention]

[0011] However, the conventional technology has the following problems. First, the problems when used as a pacemaker will be described. The method of displaying bioinformation such as heart rate in numbers, as in the technology described in Patent Documents 1 to 3, is not only not suitable for the user to intuitively recognize his / her own pace, but also requires the user to continuously check the measurement results in order to feed back the measured bioinformation to the user and control the exercise load. This not only increases the time the user spends looking at the display unit, but also blocks the view from the mobile terminal, which may lead to unexpected accidents due to carelessness of the feet and surroundings. Even if the user can check bioinformation such as heart rate, if the user tries to speed up or slow down his / her own pace based on the results, it is difficult for the methods described in Patent Documents 4 to 7 to intuitively and in real time to recognize whether the user's moving speed is fast or slow, or how much faster or slower it is. In addition, moving while looking at a mobile terminal is boring and lacks fun.

[0012] On the other hand, walking with pets or other people is more enjoyable and easier to continue than walking alone, but keeping a pet in reality is hard work, and when walking with other people, it can be difficult to maintain one's own pace suitable for aerobic exercise if there is a difference in physical strength.

[0013] Next, problems when used as a portable navigation system will be described. The method of displaying a map as in the technology described in Patent Document 4 or Patent Document 6, and the method of displaying an arrow or the like as in the technology described in Patent Document 5 or Patent Document 7 require an operation to return to the guide route when the direction of travel is lost. In addition, in the case of the method of displaying the self-position on a map, it is difficult for children who are insufficient in map reading ability and visually impaired people to intuitively recognize the direction of travel. Furthermore, when traveling a long distance, in order to reach the destination without giving up on the way, pacing or marching plan such as which section to travel and how much time to travel is essential. However, conventional navigation systems were not capable of presenting the optimal travel pace or planned travel speed in real time. Therefore, the present invention aims to solve such problems of the conventional technology and provide a virtual space travel device that uses augmented reality technology to function as a pacemaker for continuing exercise while having fun under normal circumstances. [Means for solving the problem]

[0014] That is, the virtual space movement device of the first aspect comprises a memory unit that stores movement course information including the scenery when a character moves in the virtual space, an acquisition unit that acquires bioinformation indicating the biometric condition of an exercising user and exercise amount information indicating the amount of exercise of the user, a calculation unit that calculates movement quality information indicating the quality of the user's movement and the quality of movement of a character linked to the user based on the bioinformation and the exercise amount information, and a control unit that controls the movement course information by changing the viewpoint in the virtual space based on the movement quality information.

[0015] In a second aspect, in the virtual space movement device according to the first aspect, the movement quality information is an evaluation of the user's movement state and the character's movement state, and the control unit may control the movement course information based on the information on the user's movement state and the character's movement state.

[0016] A third aspect is a virtual space movement device according to the second aspect, wherein the virtual space movement device is provided with an output unit that outputs movement course information and a character, and the control unit may change the viewpoint and change light source settings of the movement course information output to the output unit. As for a fourth aspect, in the virtual space movement device according to the third aspect, the control unit may change the viewpoint and control sound effects output to the output unit.

[0017] In a fifth aspect, the virtual space movement device may be equipped with shoes that deliver impact to the soles of the user's feet, and the control unit may control the type, magnitude, and timing of the shoe impact based on the movement quality information and the movement course information.

[0018] In a sixth aspect, the output unit is an odor generating device that generates multiple types of odors, and the odor generating device may select and emit a specified odor from the multiple types of odors based on the movement quality information.

[0019] In a seventh aspect, in the virtual space movement device according to the third aspect, the virtual space movement device includes a purchasing unit that makes it possible to purchase provable accessory information by a non-fungible token attached to a character, and the memory unit includes at least Exercise information The control unit may perform control so that the stored accessory information is attached to the character, superimposed on the movement course information, and output by the output unit.

[0020] In an eighth aspect, in the virtual space moving device according to the seventh aspect, the purchasing unit enables the user to update a smart contract and issues an instruction to store the updated smart contract in the memory unit, and the output unit may output the smart contract stored in the memory unit.

[0021] A ninth aspect may be such that, in the virtual space movement device according to the third aspect, a direction indication unit is provided that indicates the movement direction to the user via the output unit based on movement course information about the movement of the character.

[0022] A tenth aspect is a virtual space movement device according to the seventh aspect, wherein the virtual space movement device further includes a connection unit that connects the virtual space with another virtual space, and character or accessory information is output in the other virtual space, and the memory unit may store as historical information that the character or accessory information has been used in the other virtual space.

[0023] A virtual space movement method according to an eleventh aspect includes a storage step of a computer storing movement course information including scenery when a character moves in a virtual space. 、 Shows the user's activity level Exercise information and an acquisition step of acquiring Exercise information Based on We evaluated the user's motion and the movement of the character that moves in the virtual space in conjunction with the user. A calculation step of calculating movement quality information and a control step of controlling movement course information by changing the viewpoint in the virtual space based on the movement quality information are executed.

[0024] A virtual space movement program according to a twelfth aspect of the present invention includes a memory function for storing movement course information including scenery when a character moves in a virtual space in a computer. ,before Shows the user's exercise volume Exercise information An acquisition function to acquire Exercise information Based on We evaluated the user's motion and the movement of the character that moves in the virtual space in conjunction with the user. A calculation function for calculating movement quality information, and a control function for controlling movement course information by changing a viewpoint in a virtual space based on the movement quality information, realization To make. Effect of the Invention

[0025] The virtual space movement device and the like according to the present disclosure include a storage unit that stores movement course information including scenery when a character moves in a virtual space; , Yu Shows the momentum of the user Exercise information An acquisition unit for acquiring Exercise information Based on We evaluated the user's motion and the movement of the character that moves in the virtual space in conjunction with the user. The device includes a calculation unit for calculating movement quality information and a control unit for controlling movement course information by changing the viewpoint in the virtual space based on the movement quality information, so that the user can exercise continuously and enjoyably without getting bored. [Brief description of the drawings]

[0026] [Figure 1] FIG. 2 is a block diagram for explaining the functional configuration of the virtual space movement device according to the present embodiment. [Diagram 2] 4 is a flowchart of a virtual space movement program according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] The functional configuration of a virtual space movement device 10 according to this embodiment will be described with reference to Fig. 1. The virtual space movement device 10 includes a storage unit 100, an acquisition unit 200, a calculation unit 300, a control unit 400, an output unit 500, a purchase unit 600, a direction indication unit 700, and a connection unit 800. The storage unit 100 stores movement course information 14 including the scenery when the character 13 moves in the virtual space. The acquisition unit 200 acquires biological information indicating a biological condition of a user who is exercising and exercise amount information indicating an amount of exercise of the user. (exercise information) Get the.

[0028] The calculation unit 300 calculates the biological information and the exercise amount information. (exercise information) Based on this, movement quality information is calculated that indicates the quality of the user's movement and the quality of the movement of the character 13 that is linked to the user. The movement quality information is an evaluation of the user's motion state and the character 13's movement state. The control unit 400 controls the movement course information 14 by changing the viewpoint in the virtual space based on the movement quality information. The control unit 400 controls the movement course information 14 based on information on the user's motion state and the character's 13 movement state. The output unit 500 outputs the movement course information 14 and the character 13 . The control unit 400 changes the viewpoint and also changes the light source setting of the movement course information 14 outputted to the output unit 500 . The control unit 400 changes the viewpoint and also controls the sound effects output to the output unit 500 .

[0029] The virtual space movement device 10 is equipped with shoes 11 that apply impacts to the soles of the user's feet, and the control unit 400 controls the type, magnitude, and timing of the impact of the shoes 11 based on the movement quality information and movement course information 14.

[0030] The output unit 500 is an odor generation device 12 that generates a plurality of types of odors, and the odor generation device 12 selects and generates a predetermined odor from the plurality of types of odors based on the movement quality information.

[0031] The virtual space movement device 10 includes a purchasing unit 600 that allows the purchase of provable accessory information by a non-fungible token attached to the character 13, and the storage unit 100 stores at least locomotion information (exercise information) The control unit 400 controls the output unit 500 to attach the stored accessory information to the character 13, superimpose it on the movement course information 14, and output it. The purchasing unit 600 enables the user to update the smart contract he or she has signed up for, and issues a command to store the updated smart contract in the memory unit 100. The output unit 500 outputs the smart contract stored in the memory unit 100. The direction indication unit 700 indicates the movement direction to the user via the output unit 500 based on the movement course information of the character's movement. The connection unit 800 connects a virtual space to another virtual space.

[0032] The character 13 or accessory information is output in another virtual space, and the storage unit 100 stores, as history information, the fact that the character 13 or accessory information 15 has been used in the other virtual space.

[0033] Next, a virtual space movement method according to this embodiment will be described together with a virtual space movement program with reference to Fig. 2. Fig. 2 is a flowchart of the virtual space movement program according to this embodiment.

[0034] As shown in FIG. 2, the virtual space movement program includes a storage step S100, an acquisition step S200, a calculation step S300, a control step S400, an output step S500, a purchase step S600, a direction indication step S700, and a connection step S800.

[0035] The virtual space movement device 10 loads an operating space movement program stored in a ROM (not shown) or a storage unit 100 into the main memory, and executes the virtual space movement program by a CPU (not shown).

[0036] The virtual space movement program causes the CPU of the virtual space movement device 10 to realize functions such as a storage function, an acquisition function, a calculation function, a control function, an output function, a purchase function, a direction indication function, and a connection function.

[0037] Although the above-mentioned functions are processed in the order shown in the flowchart of FIG. 2, the present invention is not limited to this order, and the virtual space movement program may be executed by appropriately changing the order of the functions.

[0038] Note that the above-mentioned functions overlap with the explanations of the memory unit 100, acquisition unit 200, calculation unit 300, control unit 400, output unit 500, purchase unit 600, direction indication unit 700, and connection unit 800 of the virtual space movement device 10, and therefore detailed explanations thereof will be omitted. The storage function stores information about a course of movement including the scenery when a character moves within a virtual space (S100: storage step). The acquisition function acquires biological information indicating the biological condition of the user who is exercising and exercise amount information indicating the amount of exercise of the user (S200: acquisition step).

[0039] The calculation function is biometric and exercise information (exercise information) Based on this, movement quality information is calculated that indicates the quality of the user's movement and the quality of the movement of a character linked to the user (S300: calculation step). The control function controls the movement course information by changing the viewpoint in the virtual space based on the movement quality information (S400: control step). The output function outputs the movement course information and the character (S500: output step). The purchasing function enables the purchase of provable accessory information by a non-fungible token attached to a character (S500: purchasing step). The direction instruction function instructs the user on the direction of movement via the output function based on the movement course information of the character (S700: direction instruction step). The linking function links a virtual space with another virtual space (S800: linking step).

[0040] The virtual space movement device according to this embodiment will be described below. In this embodiment, the user selects a walking course in the virtual space. A sensor attached to the body acquires motion information or bio-information when the user actually walks. The motion information includes pace, number of steps, posture, heart rate, ground reaction force, electromyographic signals, joint accuracy, acceleration, etc. A virtual transformation distance in the virtual environment is calculated from the motion information acquired by the sensor. The virtual transformation distance is calculated and corrected for walking quality based on the bio-information. A virtual camera in the virtual space is moved from a first position to a second position according to the virtual transformation distance. An image acquired by the virtual camera is displayed on a display such as a smartphone.

[0041] In this embodiment, the camera and light source in the virtual space move based on the motion information or bio-information. This movement may be controlled to change the sound. The camera angle, light source (lighting), and sound effects are changed according to the user's actions.

[0042] In this embodiment, the conditions of the virtual space change depending on the product (shoes, caps, etc.). For example, you can run on a new course, in a new season, or at a new time of day, allowing you to experience a different virtual space depending on the shoes you wear. By combining shoes and caps, you can run on your favorite course in your favorite season. You can choose shoes or caps as if you were acquiring an item by paying for it, and experience walking.

[0043] In this embodiment, the route guided in the real world is changed according to the road conditions in the virtual space. For example, when crossing the mountains of Hakone in the virtual space, the driver is guided (introduced) to a tough route in the real world. The driver cannot clear the route unless he or she passes through the tough route.

[0044] In this embodiment, the walking speed is changed according to the road conditions in the virtual space. For example, when walking up a slope in the virtual space, the speed and angle of the treadmill are increased to provide a load equivalent to that of walking up a slope. Rhythmic sounds and weak electric currents are also acceptable. The tempo of the sound and stimulation is increased. The number of steps and thigh-raising movements (that can be cleared) is changed according to the road conditions in the virtual space. A load commensurate with the calories burned on the road conditions in the virtual space is imposed on the user in the real world.

[0045] In view of the fact that telecommuting has become mainstream, this embodiment senses the real world and changes the virtual space even when the user jogs, lifts their legs, moves around at home, etc. The movement of going up and down the stairs at home is converted into movement, action, etc. in the game in the virtual space. Going up and down the stairs at home becomes fun.

[0046] In this embodiment, you take a walk with a character that resembles a pet (dog, cat) in a virtual space. Your pet grows as you take a walk (exercise). You can raise a pet in a virtual space. An entity that is not your reflection (avatar), a supportive object (pet, etc.) walks together. You walk, stop, take detours, rest, feed, run, etc. for this entity. You walk with an entity that restricts your actions.

[0047] In this embodiment, a virtual space is created that utilizes the human tendency to walk. For example, a golf course, a game of tag, a walk with a child, rescuing a person in distress, or a walk with your child in the virtual space. Walking and running while chasing your child. In this embodiment, a virtual space is created using images taken by the user. The virtual space is created as the user walks, and as the number of users increases, the virtual space is updated.

[0048] In this embodiment, a virtual space is created by stitching together photos of the route the user walked. The photos may be those of a third party (for example, photos from Google (registered trademark) Maps) or may be photos taken by the user. The course the user actually walked is stored in GPS and displayed on Google Maps. The scenery is initially set, but changes to the scenery can be controlled by walking. The scenery changes as the number of participants increases.

[0049] In this embodiment, rather than creating a virtual space, it is created regardless of one's will. When there are few users in the virtual space, it is unexplored territory, but as the number of users increases, the virtual space is created by the photos brought by users and the photos selected by users. The scenery and landscape change according to the actions and behavior of the user.

[0050] In this embodiment, a virtual space is constructed beyond one's control, but one actively constructs the virtual space by cooperating with others of one's own will. Basically, a virtual space is constructed from one's own worldview (photos one has taken, selected), but by collaborating with others, one's worldview (photos one has taken, selected) is integrated and mixed with the other person's to construct the virtual space. A virtual space is created through collaboration. A walking worldview (virtual space) is created through collaboration. One can bring together photos, videos, and music to create a virtual space to walk in.

[0051] In this embodiment, you walk in a virtual space that you have created by yourself or with others (collaboration). You walk while creating it. You take a photo every day and create a virtual space with those photos. You walk in a virtual world based on a diary. By walking in this virtual space, you can reminisce about your past. You can walk while reminiscing about the world of others, your partner, your parents, and your children.

[0052] In this embodiment, a walking course in the virtual world is created by walking in the real world. A walking course in the virtual space is created based on the video while walking in the real world.

[0053] In this embodiment, we want to link the purchased shoes to the quality of walking. A sensor that works with an app is built into the shoes. A pressure sensor is built into the sole of the shoes on a mountain climbing course (virtual space) to measure the load on the sole of the foot when climbing to the summit. A sensor that measures the pulling force is built into the shoelaces to measure the force of pushing down when climbing to the summit. The load on the ankles, knees, etc. is measured.

[0054] In this embodiment, the situation in the virtual space changes depending on the type of shoes. If you buy business shoes, you can walk in the virtual world of Marunouchi and Kasumigaseki. If you buy golf shoes, you can walk on the golf course. If you buy running shoes for the gym, you can walk inside the gym facilities.

[0055] In this embodiment, shoes linked to virtual space data are sold at an online shop. A purchaser purchases the shoes for the purpose of walking in a new virtual space.

[0056] Also, a shared platform for running a game using a virtual space may be provided separately, and the virtual space movement of this embodiment may be incorporated into that game.

[0057] In this embodiment, after hard exercise, the user can select shoes, clothing, etc. For example, after high-stress exercise or activity, or after running a lot, the user can purchase items such as shoes and clothing in the game. The types of items that can be selected change depending on the amount of exercise strain. If the user succeeds in dieting and reaches the goal, the user can make a payment. For example, in a "walking game," the user can improve walking and running skills by meeting bosses and national ranking athletes while walking, and the user can proceed through the game advantageously and efficiently. The user is not forced to skip walking, and is given the opportunity to enjoy the game advantageously and efficiently. A mechanism that changes game settings according to the amount of strain, and gives coins according to the amount of walking.

[0058] For example, measures are needed to anticipate accidents such as bumping into someone while lifting your legs. It would be good to have devices, mechanisms, and methods to suppress and prevent users from becoming too engrossed in the virtual space and causing trouble in the real world. Give users a reason (purpose) to walk. The general idea is to "maintain health," but an easy way is to change their mood and reset their mind. With remote work becoming more widespread, many users are looking for a change of mood and a way to reset themselves.

[0059] This technology displays AR tags representing disaster information and evacuation routes, and was able to provide appropriate disaster prevention information in real time to users in a small area, but the evacuation route is displayed as an AR tag of an arrow, and a guide such as an electronic pet or a virtual person is displayed, and furthermore, this guide is displayed by moving away or approaching depending on the user's heart rate and moving speed, so that the user can intuitively recognize the evacuation route and the optimal moving speed. This technology differs from the present embodiment in the features of the technical idea of ​​the invention (problems to be solved, means for solving the problem, and effects) in that the technology is different from the present embodiment in that the guide is displayed by moving away or approaching depending on the user's heart rate and moving speed, so that the user can intuitively recognize the evacuation route and the optimal moving speed. Also, according to the activity measurement system of the fifth embodiment, the server transmits warning information to an information terminal of a user who may have fallen, and the information terminal that receives the warning information sounds an alarm, so that if the user actually falls and is injured, a person near the user (who may not be a user of the activity measurement system) can notice and rescue the user by not canceling the alarm. Also, for example, when a user moves in a virtual vehicle in a virtual space, the user does not move while the virtual space is displayed, but moves in the virtual space by driving and operating the vehicle, and a trigger occurs when the user passes a boundary line or a boundary surface by moving in the virtual space, a display transition is executed, and the display of the real space is started. Similarly, in this embodiment, the server determines whether the physical condition level of each user is at a danger level or not, and transmits warning information to users who are at the danger level or who may reach the danger level within a predetermined time, but the server may determine the physical condition level of each user at a multiple level (for example, three levels of normal level, caution level, and danger level) and transmit warning information to users who have reached a predetermined level (for example, caution level) that does not reach the danger level or who may reach the predetermined level within a predetermined time.When the user turns his / her body, the room door reappears in the user's field of view, and what the user sees through the room door is the real screen outside the room; that is, in this embodiment, when the current terminal position does not pass through the spatial region again and the current field of view covers the virtual entry due to movement, a field of view located within the virtual entry in the current field of view is determined, and the screen within the field of view determined in the acquired image frame is displayed in the virtual entry.

[0060] Here, the control unit may determine the timing to execute a human error induction program that attempts to induce mistakes due to frustration (stress) based on the GSR detection signal and the heart rate detection signal output by the GSR detection unit and the heart rate detector as bioinformation detectors, or may select whether to move the virtual body toward the dirt less than intended by the operator, make the dirt harder to remove, display a message that will frustrated the operator (stress the operator), or execute a combination of these.

[0061] In addition, when the server device determines the character image to be displayed based on the user's situation determined from the bio-information, the exercise information, and the diet information, the server device may transmit information specifying the character image to be displayed and information specifying the expression and the movement of the character to be displayed to the bio-kinetic information display device, instead of transmitting the character image data itself to be displayed. In addition, when the server device determines the character image to be displayed based on the user's situation determined from the bio-information / exercise information, the server device may transmit information specifying the character image to be displayed and information specifying the expression and the movement of the character to be displayed to the bio-kinetic information display device, instead of transmitting the character image data itself to be displayed.

[0062] So far, we have explained the operation of displaying a character image according to the user's own current situation and reproducing and displaying a character image showing a past situation, but next we will explain a bio-kinetic information processing system that allows the user to know the situation of others through a character image and the device that constitutes the system. Furthermore, if the detected bio-information such as heart rate and blood pressure is a normal value, it is not used to display a character, and only if it becomes a physically abnormal value, the bio-information is used as the subject of the character display, so that a character display that has the meaning of a warning or caution for oneself or others is executed.

[0063] In one example of this embodiment, it is not necessarily necessary for the character posture setting unit to determine the position coordinates in the virtual space of all of the above-mentioned nodes of the virtual character using the positions in real space of the user's body parts and a scaling coefficient. Instead, the position in virtual space of a pre-selected selected node from among the nodes included in the bone model of the virtual character may be determined using the estimated positions in real space of the user's body parts and a scaling coefficient, and then the positions of each node included in the bone model of the virtual character other than the selected node may be determined based on the position of the selected node in virtual space that has been determined.

[0064] In the physical function training method of this embodiment, for example, the movement information conversion process may include a current position measurement process and a training position information generation process, in which the current position measurement process measures position information of the entire or part of the user's body from the acquired movement information, the training position information generation process calculates a current speed from the position information and generates training position information by adding / subtracting a set speed from the current speed, the training movement information in the movement information conversion process includes training position information, and the image information generation process may generate training virtual space image information that reflects the training position information on the virtual space.

[0065] The patent document proposes a wrist-worn environmental data measuring device having a case to which a wrist band is connected, a temperature sensor for detecting temperature, and a display control means for displaying the temperature on a predetermined area of ​​a display section formed on the surface of the case based on the detection result of the temperature sensor, the temperature sensor being arranged in a state in which its heat-sensitive section protrudes from the case. Even if the weather measuring device does not have a wind speed sensor, the activity amount calculation section can calculate the pressure gradient (= pressure difference between two points / distance between two points) between two points (between two weather measuring devices) from the pressure data and estimate the current wind speed from the distribution of the pressure gradient, so that the activity amount of each user may be corrected according to the estimated current wind speed. The biological information is, for example, information such as heart rate, pulse rate, respiratory rate, blood glucose level, sweat rate, body temperature, and salinity concentration in blood, and the biological information sensor is a sensor that acquires at least a part of the biological information among these various biological information, and is worn in a position where it can acquire the necessary biological information from the user.

[0066] Here, GX, GY, and GZ are digital values ​​(angular velocity information) corresponding to the direction and magnitude of the angular velocity around the X-axis, Y-axis, and Z-axis applied to the sensor module, respectively, and AX, AY, and AZ are digital values ​​(acceleration information) corresponding to the direction and magnitude of the acceleration in the X-axis, Y-axis, and Z-axis directions applied to the sensor module, respectively.

[0067] For example, if biometric information about a user of a biometric information transmission device is stored for a certain period of time in a biometric information database of a server device, it is possible to specify a past point in time within the stored range and extract the biometric information, thereby displaying the past condition of another person using a character image.

[0068] In addition, when the server device determines the character image to be displayed based on the user's condition determined from bio-information, exercise information, and dietary information, the server device may transmit, rather than transmitting the character image data to be displayed itself, information specifying the character image to be displayed and information specifying the expression and movement of the character to be displayed to the bio-exercise information display device.

[0069] In other words, in a system in which the bio-kinetic information transmitting device and the bio-kinetic information display device communicate directly, the bio-kinetic information display device can receive bio-information, exercise information, dietary information, location information, etc. detected by the bio-kinetic information transmitting device, making it possible to display characters based on other people's bio-information, exercise information, and dietary information, or to display characters on a map image using location information.

[0070] In addition, when the server device determines the character image to be displayed based on the user's situation determined from the bio-information / kinetic information, the server device may transmit information specifying the character image to be displayed or information specifying the facial expression and movement of the character to be displayed to the bio-kinetic information display device, rather than transmitting the character image data itself to be displayed. That is, in a system in which the bio-kinetic information transmitting device and the bio-kinetic information display device directly communicate with each other, the bio-kinetic information display device can receive bio-information, kinetic information, position information, etc. detected by the bio-kinetic information transmitting device, and thereby display a character based on the bio-information and kinetic information of another person, or display a character on a map image using position information.

[0071] Here, the target pace extracted from the target pace database may be generated based on the latest running distance and running time from among the training data of running that the user has done in the past, or, in a case where the target pace database stores the above-mentioned training data in association with topographical information of the running course (elevation, course shape, etc.) and meteorological information at the time (temperature, wind direction, wind speed, etc.), the target pace may be generated based on the running distance and running time extracted based on various conditions such as the topographical information and meteorological information input by the user.

[0072] In each of the above operation modes, when the user temporarily stops exercising (running) for a rest, water intake, or the like, and the detected running speed drops (slows down), the image of the virtual runner may be temporarily stopped, for example, by the user operating the operation unit, so that the virtual runner does not increase its running speed (pace up) and move away based on the above processing operation. When it is determined in step that the pacemaker virtual object has reached the right edge of the display screen (determination of the right edge of the screen display), the arithmetic circuit replaces the target pace set for the running section with the user's current running pace (actual pace) calculated based on GPS data, and displays it in the target pace display on the display screen.

[0073] For example, when a decrease in the user's pitch is detected, when a decrease in the user's pace is detected, or at any timing, a moving image may be generated and displayed in which the time from foot lift-off to landing of the virtual runner is slightly faster than the user's pitch, i.e., the pitch is slightly faster, to encourage the user to increase the pitch. Also, it has been very difficult to secure personnel who meet the capabilities and conditions required for a pacemaker, that is, a runner with the same or greater running ability as the user and who can understand the user's ability and provide appropriate guidance, but by applying the exercise support device and exercise support method described above, a pacemaker that is optimal for oneself can be easily used.

[0074] For example, when the visual virtual agent determines that the user is running, the user's activity data indicates that the user is running, and the activity data can be acquired using a sensor such as a GPS tracker or a step sensor, and the visual virtual agent may be further configured to acquire user-specific data, for example, from the GPS tracker or the step sensor, and thus may be configured to monitor and automatically determine whether the user has stopped running. The distance traveled when not participating in a game may be measured by a terminal device or a pedometer, and the number of games to be played, the distance traveled, etc., may be calculated based on the distance traveled and the distance traveled during the game in order to consume a target amount of calories.

[0075] In this case, the pitch set for the virtual runner may be faster than the user's pitch by a fixed number of steps across the board, or faster by a fixed ratio of steps depending on the user's pitch, or may be variably set based on other calculation formulas, conditional formulas, etc. The motion measurement unit may measure the distance traveled using GPS, or may calculate the distance traveled by measuring the number of steps and multiplying the number of steps by the stride length. The management unit may change the size of the character image depending on the movement history, for example, the distance, movement speed, number of steps, etc., traveled by the user in real space.

[0076] In addition, in the above embodiment, in the virtual camera interval determination process, the interval between the outer imaging unit (left) and the outer imaging unit (right) in the marker coordinate system is calculated, and then the position and orientation of either the left virtual camera or the right virtual camera is determined from the position and orientation of the other virtual camera calculated based on the marker recognition result based on the interval. However, in other embodiments, the position and orientation of the outer imaging unit (for example, the average position of the position of the outer imaging unit (right) and the position of the outer imaging unit (left), the average orientation of the orientation of the outer imaging unit (right) and the orientation of the outer imaging unit (left)) may be calculated based on the position and orientation of the outer imaging unit (left) calculated based on the marker recognition result for the left real world image, and the position and orientation of the outer imaging unit (right) calculated based on the marker recognition result for the right real world image, and the positions and / or orientations of the left virtual camera and right virtual camera may be determined based on this. In one example of this embodiment, it is not necessarily necessary for the character posture setting unit to determine the position coordinates in the virtual space of all of the above-mentioned nodes of the virtual character using the positions in real space of the user's body parts and a scaling coefficient. Instead, the position in virtual space of a pre-selected selected node from among the nodes included in the bone model of the virtual character may be determined using the estimated positions in real space of the user's body parts and a scaling coefficient, and then the positions of each node included in the bone model of the virtual character other than the selected node may be determined based on the position of the selected node in virtual space that has been determined.

[0077] Alternatively, the trained model may generate time series data as learning data by having the trainer tag (comments on how to move body parts in an ideal way, that the trainee should breathe, that the trainee's heart rate is too high and the trainee should take a break, etc.) the time series data that is the result of measuring the trainee's body movements (point cloud data, photographed images of the trainee).

[0078] Also, in the case of calculating the assumed position from the assumed moving speed of the guide as in the portable movement support device described in the claims, the heart rate meter 9 is used to determine the assumed moving speed, and if the user's heart rate is higher than the target heart rate, the assumed moving speed of the guide is slowed down, and if it is lower, the assumed moving speed of the guide is accelerated. Here, "based on the GSR detection signal and the heart rate detection signal" means, for example, executing a human error inducement program in response to the change in the numerical value indicated by the GSR detection signal and the heart rate detection signal crossing a predetermined threshold, or setting the specific content of the human error inducement program to be executed in advance in response to the numerical value or the change in the numerical value indicated by the GSR detection signal and the heart rate detection signal, and executing the content. Exercise intensity is an index that indicates the degree of intensity of a person's exercise state (movement state), and may be, for example, an index corresponding to a person's moving speed, vibration, heart rate, body temperature, amount of sweat, etc., or may be subjective exercise intensity (an index that indicates the intensity of exercise from a psychological aspect of a person).

[0079] Here, the game situation is, for example, the situation of the user's game play, the progress of the game, the situation of events in the game, the battle situation of the game, the match situation of the game, the attack situation of the enemy appearing in the game, the hit judgment situation in the game, the situation of the moving body (character or hit object, etc.) appearing in the game, the situation of the character or user's status parameters (stamina value, level, attack power or defense power, etc.), or the situation of the game results, and is, for example, various situations expressed by game parameters, etc. For example, the output control unit can output a practice menu selected by the processing unit as recommended information, and more specifically, it controls the display unit (display device) of the feedback device to display a predetermined image, controls a force sense mechanism (wearable device) attached to the user's body to provide tactile stimulation to the user, or controls an audio output unit (audio output device) to output a predetermined sound. In addition, if a user is wearing a device capable of presenting a haptic sensation on a hand or the like, the information processing device may instruct the device to present a predetermined haptic sensation when a target virtual object in the virtual space comes into contact with the user's finger or when the contact state between the user's finger and the target virtual object is maintained.

[0080] Physical quantities defined in physics such as Newtonian mechanics and thermodynamics include, for example, force (F = mass x acceleration), momentum (P = mass x velocity), work (W = force F x distance traveled), kinetic energy (K = 1 / 2 x mass x speed 2), heat (J = 4.184 x kinetic energy), velocity (rate of change of position of living body per unit time), acceleration (rate of change of velocity per unit time), acceleration rate (jerk: rate of change of acceleration per unit time), etc. Here, the time derivative of angular momentum is torque, and the time derivative of momentum is force, and in order to continuously generate torque and force in a certain direction, it is necessary to continuously accelerate the rotation speed of a motor or a linear motor, and therefore a method of periodically rotating a rotating body is not suitable for continuously presenting a force sensation in a certain direction.

[0081] The reference operation instruction direction may be calculated by correcting the acceleration indicated by the acceleration data, or may be calculated by correcting the direction of magnetism detected by the magnetic sensor, or may be calculated by having the user perform a predetermined operation with the terminal device in a specific posture (e.g., placed on a table and stationary) and assuming that the x-axis direction and the y-axis direction at the time the predetermined operation is performed are horizontal. The data may include data captured from a sensor (e.g., operably coupled to the frame or otherwise attached to the user), such as an image capture device (e.g., a camera in an inward-facing imaging system and / or an outward-facing imaging system), an audio sensor (e.g., a microphone), an inertial measurement unit, an accelerometer, a compass, a global positioning system unit, a wireless device, or a gyroscope, or data obtained or processed using a remote processing module or a remote data repository, possibly for processing or reading and then passing to the display. The data may include data captured from sensors (e.g., that may be operatively coupled to the frame or otherwise attached to the user), such as image capture devices (e.g., cameras in the inward-facing imaging system and / or the outward-facing imaging system), microphones, inertial measurement units, accelerometers, compasses, global positioning system units, wireless devices, or gyroscopes, or data obtained or processed using a remote processing module or remote data repository, possibly for transmission to the display after such processing or readout.

[0082] The virtual environment may be used to simulate a real environment in reality, for example, the virtual environment may include sky, land, or sea, the land may include environmental elements such as a desert or a city, the user may control virtual objects to move in the virtual environment, the virtual objects may be virtual avatars representing the user in the virtual environment, the virtual avatars may take any form, such as a human or animal, and the application is not limited thereto.

[0083] In one possible embodiment, the first electronic device is an electronic device capable of displaying a virtual environment, i.e., an electronic device equipped with a game application, and each first electronic device can control at least one virtual object, and multiple first electronic devices can control multiple virtual objects, and a user can control through the first electronic device such that the virtual objects fight in the virtual environment, and this first electronic device may be an electronic device such as a mobile phone, a tablet, a laptop, etc., and this first electronic device may be equipped with a Bluetooth module used to establish a connection with a Bluetooth module of another electronic device. In order to solve the above problems, the virtual environment construction device is a virtual environment construction device that constructs a virtual environment for a user to view based on a real environment in which a plurality of objects exist, and includes: a model construction method determination unit that determines a model construction method indicating whether to model the object using CG or live-action video based on at least one of the sensing difficulty and CG drawing difficulty of the target object, for at least one of the plurality of objects as a target object; a state sensing unit that measures the state of the target object in the real environment; an environmental model construction unit that constructs an environmental model for drawing in the virtual environment for static objects that do not change over time according to the model construction method; and a dynamic object construction unit that constructs a model for drawing in the virtual environment for dynamic objects that change over time according to the model construction method and synthesizes it into the environmental model. The virtual object is a virtual object controlled by the first electronic device, the virtual object may obtain at least one virtual item in the virtual environment, the virtual items may be bound to an account of the virtual object, the virtual object may indicate that it has the virtual items, the virtual object may use or discard the virtual items, and the first electronic device transfers the virtual item to the virtual environment by the virtual object discarding the virtual item in the virtual environment.

[0084] In the factory calibration module, some of the information required to calculate the transformation applied to the image / video may be generated during factory calibration or may be calculated based on further information on the actual camera orientation at the site, such as height above a floor or desk, 3D viewpoint, field of view of the lens, etc. As described above, when the controller is tilted beyond the limit of the virtual camera's range of motion and then the pitch angle of the controller is returned, the pitch rate may be changed according to the pitch angle of the controller according to this changed correspondence until the pitch rate of the virtual camera becomes 0, or the correspondence may be gradually returned to the original correspondence. Also, in the above example, as the pitch angle of the controller falls below the change limit and becomes smaller, the correspondence gradually changes to approach the original correspondence, but instead, the correspondence may be gradually changed to approach the original correspondence according to the rate of decrease of the pitch angle of the controller.

[0085] The short-term feedback may include visual indications of the quality and completion of the repetitions, such as a scoreboard that records the number of repetitions, a virtual character that mimics or reflects the player's movements, animation of the virtual character that cheers, sings, plays music, and / or otherwise motivates the player to continue, and physical changes in the virtual reality environment, such as the sun rising in the virtual reality environment, food growing, ice melting, wind blowing, or a construction project (initiated by a villager supported by the player). The exercise evaluation unit may also compare the accumulated exercise calories with the total calories consumed by the user within a predetermined period (e.g., during one day), or may compare the accumulated exercise calories with the carbohydrate calories consumed by the user within a predetermined period. For example, the exercise time setting unit may calculate the target exercise time as described above for a user with a diabetic constitution, using carbohydrate calories as calories to be consumed, and for other users, may calculate the target exercise time using total calories as calories to be consumed. The control element module may include one or more of the following features: control and management of camera settings to optimize quality, configuration of other hardware elements, communication between the algorithm module and higher-level code, applications, and user interfaces, and pushing calibration data from the factory to the algorithm element. The total calories and carbohydrate calories may be calculated by the user information acquisition unit based on user information, or may be calculated by another device (e.g., a wearable terminal, a server, etc.).

[0086] A display control device according to one aspect of the present embodiment is a display control device that controls display of a head mounted display having a display unit capable of displaying a virtual space image representing a virtual space seen in a viewing direction from a virtual viewpoint in the virtual space as a stereoscopic image utilizing binocular parallax, the display control device including: a detection result acquisition means for acquiring a detection result of an orientation of the head mounted display; a means for acquiring a first virtual space image, the first acquisition means being configured to newly acquire a first virtual space image in a viewing direction corresponding to a changed orientation of the head mounted display based on a change in the orientation of the head mounted display; a means for acquiring a second virtual space image, the second acquisition means being configured to newly acquire a second virtual space image in a viewing direction corresponding to a changed orientation of the head mounted display based on the change in the orientation of the head mounted display; and a display control means for displaying the first virtual space image and the second virtual space image. The display control device includes a display control means that executes control to synthesize images and display them on the display unit, and a change means that reduces the degree of obscuration of the second virtual space image by the first virtual space image in accordance with the passage of time when the first virtual space image and the second virtual space image are image-composited and displayed on the display unit, and the display control means further includes a means for restricting new acquisition of one of the first virtual space image and the second virtual space image when the first virtual space image and the second virtual space image are image-composited and displayed on the display unit, and the display control means moves one of the virtual space images displayed on the display unit in a movement direction based on the opposite direction to the change in orientation of the head mounted display based on a change in orientation of the head mounted display, and updates the other of the first virtual space image and the second virtual space image displayed on the display unit to the newly acquired virtual space image.As described above, as one aspect of the above embodiment, an information processing device is configured to include a position information acquisition unit, a time information acquisition unit, a weather information acquisition unit, a natural light source estimation unit, a reality corresponding object information acquisition unit, a photographed image acquisition unit, a virtual photographed image generation unit, a photographed object identification unit, and a storage unit, and acquires position information of a user terminal, acquires current time information, acquires weather information for the position based on the position information of the user terminal, estimates the status of a natural light source at the position of the user terminal based on the acquired position information, time information, and weather information, acquires information on a reality corresponding object in a virtual space that is previously associated with a real object existing around the user terminal based on the position information, and photographs the real space with a camera device equipped in the user terminal. The captured image and shooting direction information obtained in the above are acquired, and a virtual camera is installed for at least one pattern that is estimated to be in approximately the same position and approximately the same shooting direction as the captured image to capture images in a virtual space. A rendering is then performed that takes into account the effect of the light source on the real corresponding object in the estimated natural light source conditions to generate a virtual captured image. Matching points between the visual features of the captured image and the virtual captured image are identified, and the objects shown in the captured image are identified based on the degree of match between the two. Therefore, by using a virtual captured image obtained by rendering in a more realistic condition that reflects the natural light source conditions, matching can be performed under conditions where conditions such as shadows, reflections, and brightness are similar, thereby improving the accuracy of tracking.

[0087] As described above, as one aspect of the above embodiment, an information processing device is configured to include a position information acquisition unit, a time information acquisition unit, a weather information acquisition unit, a natural light source estimation unit, a reality corresponding object information acquisition unit, a photographed image acquisition unit, a virtual photographed image generation unit, a photographed object identification unit, and a storage unit, and acquires position information of a user terminal, acquires current time information, acquires weather information for the position based on the position information of the user terminal, estimates the status of a natural light source at the position of the user terminal based on the acquired position information, time information, and weather information, acquires information on a reality corresponding object in a virtual space that is previously associated with a real object existing around the user terminal based on the position information, and photographs the real space with a camera device equipped in the user terminal. The captured image and shooting direction information obtained in the above are acquired, and a virtual camera is installed for at least one pattern that is estimated to be in approximately the same position and approximately the same shooting direction as the captured image to capture images in a virtual space. A rendering is then performed that takes into account the effect of the light source on the real corresponding object in the estimated natural light source conditions to generate a virtual captured image. Matching points between the visual features of the captured image and the virtual captured image are identified, and the objects shown in the captured image are identified based on the degree of match between the two. Therefore, by using a virtual captured image obtained by rendering in a more realistic condition that reflects the natural light source conditions, matching can be performed under conditions where conditions such as shadows, reflections, and brightness are similar, thereby improving the accuracy of tracking.The main virtual camera includes a left main virtual camera and a right main virtual camera, and the sub virtual cameras include a left sub virtual camera and a right sub virtual camera, and the steps include a step of updating the field of view of the left main virtual camera and the field of view of the right main virtual camera in accordance with movement of the head mounted display, the steps include a step of generating first left eye field of view image data based on the field of view of the left main virtual camera and virtual space data, and a step of generating first right eye field of view image data based on the field of view of the right main virtual camera and virtual space data, the steps include a step of determining a position and orientation of the left sub virtual camera based on a relative positional relationship between the left main virtual camera and a reflective object, and a step of determining a position and orientation of the right sub virtual camera based on the relative positional relationship between the right main virtual camera and a reflective object, and the steps include a step of generating second left eye field of view image data based on the field of view of the left sub virtual camera and virtual space data, and a step of generating second right eye field of view image data based on the field of view of the right sub virtual camera and virtual space data.

[0088] A head mounted display system comprising: a frame configured to be supported on a user's head; a display disposed on the frame, the display configured to project light associated with a virtual image into the user's eye to display virtual image content to the user and further configured to transmit light from an environment to the user's eye to provide the user with a view of a portion of the environment; a camera configured to capture images of the environment at a frame rate, the camera having a field of view and a first frame rate; and processing electronics configured to receive and process images captured by the camera, the image processing comprising: assessing relative motion between the head mounted display and one or more features in the environment, the assessment of relative motion determining whether the head mounted display has moved, is moving, or is expected to move relative to one or more features in the environment. determining, and / or determining whether one or more features in the environment have moved, are moving, or are expected to move relative to the head mounted display; determining one or more virtual image content locations in the environment corresponding to locations where renderable virtual image content will appear to the user when the locations appear in the display; comparing one or more virtual image content locations in the environment to a visibility zone that includes at least a portion of the field of view of the camera; and processing electronics including: modifying a camera's frame rate from a first frame rate to a second frame rate and / or adjusting an amount of processing on frames captured by the camera to be processed based on the assessment of relative motion between the head mounted display and one or more features in the environment and the comparison of the one or more virtual image content locations in the environment to the visibility zone.

[0089] For example, all modules of the virtual camera control system may be included in a user terminal device (e.g., a desktop computer, a notebook computer, a workstation, a PDA, a web pad (particularly, a smart pad), a mobile phone (particularly, a smartphone), etc.) (not shown) that typically requires the use and control of a virtual camera, but some modules may be included in another digital device (e.g., a three-dimensional content providing server) (not shown) other than the user terminal device that can communicate with the user terminal device. According to an aspect of the present disclosure, an information processing device includes a storage unit that stores a program executed by the information processing device in order to provide a virtual experience to an audience watching a first performance by a performer in a real space, a control unit that controls the operation of the information processing device by executing the program, and a display unit that displays the real space. Here, the basic processing of virtual reality and augmented reality will be described. A still image or a video background and object that are real-life images captured by a smartphone camera or the like are displayed by blending processing (visible / invisible / transparent) based on real map information of a real coordinate system (latitude, longitude, altitude) with still images or CG animation objects that are 3DCG (Three-dimensional computer graphics). The camera or cameras may be an integrated part of several different electronic devices of the user (e.g., smartphone, tablet, PDA, TV) or may be located within the user's other devices (e.g., refrigerator, scale, etc.) or may be located in specific locations in rooms of the user's home (e.g., living room, bedroom, and / or kitchen). The member's terminal may be, for example, a wireless communication device that ensures portability and mobility, including all types of handheld-based wireless communication devices such as Personal Communication System (PCS), Global System for Mobile communications (GSM), etc.

[0090] A head mounted display system configured to project light into a user's eye and display augmented reality image content in a field of view of the user and image at least a portion of an environment in front of a user wearing the head mounted display system, the head mounted display comprising: a frame configured to be supported on a user's head; a head mounted display disposed on the frame, the display configured to project light into the user's eye and display virtual image content in the field of view of the user, at least a portion of the display being transparent, the head mounted display being disposed at a location in front of the user's eye such that, when the user wears the frame, the transparent portion transmits light from a portion of the environment in front of the user and the head mounted display to the user's eye to provide a view of the portion of the environment in front of the user and the head mounted display; a camera configured to acquire images at a frame rate, the camera having a field of view and a first frame rate; a non-transitory computer readable medium storing computer executable instructions; and a computer-readable medium storing the computer-readable medium and the ... an electronic device in communication with a data-readable medium, the electronic device being configured to process frames captured by a camera, assessing relative motion between the head mounted display and one or more features in the environment, the assessment of relative motion including determining whether the head mounted display has moved, is moving, or is expected to move relative to one or more features in the environment and / or determining whether one or more features in the environment have moved, are moving, or are expected to move relative to the head mounted display; determining one or more locations in the environment that correspond to locations where renderable virtual image content would appear to a user if the locations appeared in the display; comparing one or more virtual image content locations in the environment to a visibility zone that includes at least a portion of the field of view of the camera; and modifying a frame rate of the camera from a first frame rate to a second frame rate based on the assessment of relative motion between the head mounted display and one or more features in the environment and the comparison of the one or more virtual image content locations in the environment to the visibility zone;and / or electronics configured to execute computer-executable instructions to adjust the amount of processing on frames captured by the camera that are processed.

[0091] Here, a plurality of light sources L that illuminate the virtual object are arranged in the virtual space, and the virtual space information includes information on the shape and material (color, texture image) of the virtual object that may be arranged in the virtual space, the position and orientation of the virtual object arranged in the virtual space, and information on the position, direction, color, and intensity of the light source. Note that any type of control of the light source may be used, and for example, the color of the light source may be changed according to an operation input, or the position and irradiation angle of the light source may be changed when the light source is a point light source, or the irradiation direction may be changed when the light source is a parallel light source. For example, a virtual light source may be set using a real image for the left eye, and a virtual object illuminated by the virtual light source set using the real image for the left eye may be captured by the left virtual camera, and a virtual light source may be set using a real image for the right eye, and a virtual object illuminated by the virtual light source set using the real image for the right eye may be captured by the right virtual camera. In this way, when experiencing a virtual apartment room using the MR system, if the room is affected by the shadow of a virtual building outside the room due to the position of the light source, a simulation that also takes sunlight into consideration can be performed by changing the color of the three-dimensional model inside the room in consideration of the influence. A simplified example of obtaining natural light source information would be to obtain information on the direction of the sun estimated from the date and time and the location information of the user terminal, and to obtain the color of the sky estimated from weather information to create a hemispherical light source that simply adds diffuse ambient color to all objects in the scene.

[0092] The term product as used herein is intended to include a computer program accessible from any computer readable device or storage medium, for example stored on a server coupled to a cloud. For example, in a product design, if a part does not check whether it is in contact with other parts, it may lead to a malfunction. For example, the object may be an item, a treasure chest, a weapon, a piece of armor, furniture, an electrical appliance, or a decoration that is placed in a real space. The computer program product includes instructions that, when executed, perform one or more methods as described above. Here, the property information of the client refers to any information related to the client, including the product information or the setting information of the client.

[0093] Regarding the size of the cap and the size of each of the buttons, if the size of the holes formed in the lower housing is the same, it is preferable that the cap is smaller, and if the sizes of the holes are not the same, they may be approximately the same size. The cap is integrally formed with a peripheral surface portion formed in a substantially cylindrical shape, a top surface portion closing the upper side of the peripheral surface portion, and a flange protruding outward from the outer periphery of the opening at the lower end side of the peripheral surface portion. The cap is preferably formed using an elastic body.

[0094] Here, the target pace extracted from the target pace database may be generated based on the latest running distance and running time of the user's past running training data, or may be generated based on the running distance and running time extracted based on various conditions such as the topographical information and weather information input by the user when the topographical information (elevation, course shape, etc.) of the running course and the weather information (temperature, wind direction, wind speed, etc.) are stored in association with each other as the training data in the target pace database. Also, the target achievement value calculation unit may correct and calculate the remaining distance or remaining time until the activity amount of each user reaches the target activity amount based on the weather data acquired by the weather data acquisition unit from N (one or more) weather measuring devices arranged ahead along each course (the movement path of each user).

[0095] In the course selection screen, check boxes are provided to the right of "Course A", "Course B", and "Free Course". The user moves the cursor to one of the check boxes and presses the switch / determine button, and then moves the cursor to the button and presses the switch / determine button to select the checked course. In the first and second examples shown in this embodiment, as in the first embodiment described above, GPS data acquired during running by a series of exercise support methods, topographical information and weather information of the running course, etc. may be stored as training data in a target pace database of an information processing device or a network server, and the database may be updated sequentially. Specifically, the simulation processing unit performs a simulation process in which a virtual moving body, which is provided corresponding to a moving toy and whose running characteristics are set based on the running characteristic data, runs (virtual running) according to the running control data on a virtual course in a virtual space, which is provided corresponding to the course and whose course characteristics are set based on the course data.

[0096] Also, when there are multiple panoramic images at the same position, different panoramic images may be sent according to the actual situation (season, date, time, etc.) at the time of the panoramic image request, so as to match the actual situation as much as possible. However, the position or shape may be made different depending on the time (e.g., season), and even a gap may be provided between adjacent HEXes. Similarly, the positions and numbers of live streaming towers and portals arranged in each of the multiple HEXes may be made different depending on the time (e.g., season).

[0097] On the other hand, if each viewer wants to move the gift at their own time, they may input an object generation command and move the gift at their own time. In another configuration, the display device may be temporarily fixed in position relative to another structure, but people can turn it around or pick it up and move it at their own convenience. Here, the players are loaded into a campsite on top of a small hill, with an aspiring artist on the left and a nosy neighbor on the right.

[0098] A display control device according to one aspect of the present embodiment is a display control device that controls display of a head mounted display having a display unit capable of displaying a virtual space image representing a virtual space seen in a viewing direction from a virtual viewpoint in the virtual space as a stereoscopic image utilizing binocular parallax, the display control device including: a detection result acquisition means for acquiring a detection result of an orientation of the head mounted display; a means for acquiring a first virtual space image, the first acquisition means being configured to newly acquire a first virtual space image in a viewing direction corresponding to a changed orientation of the head mounted display based on a change in the orientation of the head mounted display; a means for acquiring a second virtual space image, the second acquisition means being configured to newly acquire a second virtual space image in a viewing direction corresponding to a changed orientation of the head mounted display based on the change in the orientation of the head mounted display; and a display control means for displaying the first virtual space image and the second virtual space image. The display control device includes a display control means that executes control to synthesize images and display them on the display unit, and a change means that reduces the degree of obscuration of the second virtual space image by the first virtual space image in accordance with the passage of time when the first virtual space image and the second virtual space image are image-composited and displayed on the display unit, and the display control means further includes a means for restricting new acquisition of one of the first virtual space image and the second virtual space image when the first virtual space image and the second virtual space image are image-composited and displayed on the display unit, and the display control means moves one of the virtual space images displayed on the display unit in a movement direction based on the opposite direction to the change in orientation of the head mounted display based on a change in orientation of the head mounted display, and updates the other of the first virtual space image and the second virtual space image displayed on the display unit to the newly acquired virtual space image.) Furthermore, a display control method according to one aspect of the present embodiment is a display control method for controlling the display of a head mounted display having a display unit capable of displaying a virtual space image representing a virtual space viewed in a viewing direction from a virtual viewpoint in the virtual space as a stereoscopic image utilizing binocular parallax, the display control method including a detection result acquisition step for acquiring a detection result of an orientation of the head mounted display, a first acquisition step for acquiring a first virtual space image, a second acquisition step for acquiring a second virtual space image, a display control step for executing control for image-compositing the first virtual space image and the second virtual space image and displaying the composite image on the display unit, and a change step for reducing the degree of occlusion of the second virtual space image by the first virtual space image over time, in which, in the second acquisition step, a second virtual space image in a viewing direction corresponding to the changed orientation of the head mounted display is newly acquired based on a change in the orientation of the head mounted display, and in the display control step, the second virtual space image displayed on the display unit is updated to the newly acquired second virtual space image based on the change in the orientation of the head mounted display, and the first virtual space image displayed on the display unit is moved in a moving direction based on the opposite direction to the change in the orientation of the head mounted display.

[0099] In addition, such an equipment object may be, for example, an item obtainable in the game (for example, an item obtainable from a treasure chest, which is a type of target object), or may be a paid item that is given to a user in the game by the user in the real world through a payment made by the user. The image frame may be collected from the real world by the terminal through a built-in image collecting device or an externally connected image collecting device, and the built-in image collecting device may be a front camera or a rear camera of the terminal, and the image frame may be collected from a real scene by another device and then transmitted to the terminal. For example, when there are ups and downs in the course as shown in the figure, a skier in the real world detects a change in the direction of travel (change in acceleration, change in the direction of gravity, change in centrifugal force) using the semicircular canals, and for example, directs the line of sight in a direction that looks into the distance. When the user is located in a field (play field, simulation field, play area) in the real space (real world), the position information and direction information may be position information and direction information (position coordinates in a coordinate system with a given position as the origin, and rotation angles around each coordinate axis) in the field. The position information and directional information may be position information and directional information (position coordinates in a coordinate system with a given position as the origin, and rotation angles around each coordinate axis) on a field (play field, simulation field, play area) or on a movable housing in real space (the real world) when the user plays on that field or on a movable housing.

[0100] For example, when a given operation input is made to an item at a time, the virtual camera may be moved from the local position (meaning a position in the local coordinate system of the moving object) 342-1 of the virtual camera at the time to a given reference position in a given period of time according to a given movement route, when a given operation input is made to an item at a time, the virtual camera may be moved from the local position of the virtual camera at the time to a given reference position in a given period of time according to a given movement route, when a given operation input is made to an item at a time, the virtual camera may be moved from the local position of the virtual camera at the time to a given reference position in a given period of time according to a given movement route. The camera control data may also include data defining a movement route of a trajectory connecting a given reference position in the local coordinate system of the moving object and a target position set corresponding to an item attached to the moving object, and the first camera control unit may perform a process of moving the virtual camera to the target position along the movement route when a given operation input is made to an item attached to the moving object. Here, the camera control data is defined in a time series from the start point to the end point as shown in the figure, and if the data of the start point is defined at time and the data of the intermediate point is defined at time, the position at time can be set as the playback position to control the movement of the virtual camera along the route from the intermediate point of the movement route to the end point. The camera control data includes data corresponding to a plurality of movement routes with different trajectories for the item, and the first camera control unit may select camera control data corresponding to one of the movement routes based on the position or orientation of the moving object or the value of a predetermined game parameter, and perform the first camera control process using the selected camera control data. On the other hand, the viewpoint of the virtual camera is set to a position shifted from the reference according to the direction Pd of the player character (the direction is, for example, the forward direction or the moving direction of the player character, but may also be the attack direction by the player character) based on the point Cph(xcp, ycp, zcp) in the virtual game space determined corresponding to each of the positions Ph(xh, yh, zh) of the player character (i.e., each of the points on the route R).

[0101] Also, in a place such as a slope, the direction of gravity and the normal direction of the outer circumferential surface do not necessarily need to be parallel. In this case, the control unit may guide the user to the slope that is closest to the current position of the user. Here, the game processing unit performs various game processes such as coin (cost) acceptance process, various mode setting process, game progress process, selection screen setting process, process for determining the position and rotation angle (rotation angle around X, Y or Z axis) of an object (one or more primitive faces), process for moving an object (motion process), process for determining the position of the viewpoint (position of the virtual camera) and the line of sight angle (rotation angle of the virtual camera), process for arranging objects such as map objects in the object space, hit check process, process for calculating the game result (achievement, achievement), process for multiple players to play in a common game space, and game over process, based on operation data from the operation unit, personal data from the save information storage device, game program, etc. Furthermore, in the above embodiment, the angular difference (|60 degrees - 30 degrees| = 30 degrees) between the controller pitch angle at which the virtual camera's pitch rate is 0 (the shooting direction is horizontal) and the controller pitch angle corresponding to the upper limit of the virtual camera's range of motion was the same as the angular difference (|0 degrees - 30 degrees| = 30 degrees) between the controller pitch angle at which the virtual camera's pitch rate is 0 and the controller pitch angle corresponding to the lower limit of the virtual camera's range of motion; however, this embodiment is not limited to this, and the angular difference between the controller pitch angle at which the virtual camera's pitch rate is 0 and the controller pitch angle corresponding to the upper limit of the virtual camera's range of motion may be greater or smaller than the angular difference between the controller pitch angle at which the virtual camera's pitch rate is 0 and the controller pitch angle corresponding to the lower limit of the virtual camera's range of motion. [Another example of obtaining information on the position and tilt of the user's head] In the above explanation, information on the position and tilt of the user's head is obtained from a head direction sensor provided on the display device, but this embodiment is not limited to this. For example, the position and tilt angle of the user's head may be detected by capturing an image of the user using a camera placed at a known position in the room where the user is located, and detecting the position and orientation of a specific point that moves with the user's head, for example a specific marker that has been placed in advance on the display device worn by the user.[When bones can be detected] Also, when the control unit can estimate the positions and angles of bones and joints of an object based on image data captured by the imaging unit (without using point cloud information) (a method for estimating bones and joints based on image data is widely known, so a detailed description is omitted here), the amount of movement of each bone (information on time changes related to the position and direction of each bone) may be estimated based on the estimated position and angle, not based on the amount of movement of the point cloud. In other words, when the HMD1 is in a forward rotation state and the virtual straight line LC and the operation object Op intersect, and the HMD1 further rotates by an angle in the roll direction, the display range setting unit may slide the display content of the message Ms displayed on the bulletin board Bd by a distance according to the angle in the +ZV direction, and may not change the display content of the message Ms displayed on the bulletin board Bd when the HMD1 is in a forward rotation state and the virtual straight line LC and the operation object Op intersect, and when the HMD1 is further rotated by an angle in the reverse roll direction QX, the display range setting unit may not change the display content of the message Ms displayed on the bulletin board Bd.

[0102] The present invention further relates to a visual virtual agent configured to perform an artificial physiological color change response to a user for at least one visual virtual agent state and / or for at least one state of the user, the visual virtual agent being configured to perform a visual response to the user, the at least one visual virtual agent state and / or the state of the user being determined based on user activity data and / or on the user's physiological and psychological data, the visual virtual agent being configured to adapt the visual response to changes in the user activity data and / or the user's physiological and psychological data, the adaptation of the visual response at least partially comprising a color change of the visual virtual agent for the at least one visual virtual agent state and / or for the at least one state of the user, the user's physiological and psychological data being based on the user's current behavior (non-verbal behavior) and / or current physiological state and / or current mental state (psychological state) and / or medical state. the visual virtual agent is displayed in two or three dimensions on the display device, the visual response includes a posture and / or movement of the visual virtual agent, and the audio response includes a sound, volume, emphasis, and / or accent of the visual virtual agent based on the user's activity data and / or the user's psychophysiology; and the at least one sensor is configured to acquire the user's psychophysiological parameters by speech recognition, face recognition, pulse measurement, respiration measurement, blood pressure measurement, and / or skin electrical conductivity measurement, and the at least one sensor includes an audio-visual sensor, an activity sensor, a physiological sensor, a biometric sensor, a heart rate sensor, a blood pressure sensor / monitor, a weight scale, a motion sensor, a light sensor, a video sensor, an acoustic sensor, a blood glucose monitor, a blood oxygen saturation monitor, a moisture monitor, a skin temperature gauge / thermometer, a respiratory monitor, an EEG electrode, a bed sensor, an accelerometer, an activity sensor / tracker, a video camera, a depth sensor, a skin potential activity sensor, a portable global positioning system sensor, and a microphone.The present embodiment further relates to a visual virtual agent configured to perform an artificial physiological color change response to a user for at least one visual virtual agent state and / or for at least one state of the user, the visual virtual agent being configured to perform a visual response to the user, the at least one visual virtual agent state and / or the state of the user being determined based on activity data of the user and / or on physiological and psychological data of the user, the visual virtual agent being configured to adapt the visual response to changes in the activity data of the user and / or in the physiological and psychological data of the user, The adaptation of the visual response includes at least in part a color change of the visual virtual agent for at least one visual virtual agent state and / or at least one state of the user, the user's psychophysiological data is based on the user's current behavior (non-verbal behavior) and / or current physiological state and / or current mental state (psychological state) and / or medical state, the visual response includes posture and / or movement of the visual virtual agent, and the audio response includes sound, volume, emphasis and / or accent of the visual virtual agent based on the user's activity data and / or the user's psychophysiological state.

[0103] For example, when the walking speed suddenly increases or the user starts running, the generating unit may change the tempo of the music to an up-tempo, or may change the song itself to a song with a faster tempo, and generate the music. That is, for example, when a predetermined music is played at a tempo faster than normal, the time from the start of playback of the music to the first first timing is shorter, and the time between the first timing and the second timing is also shorter.

[0104] Here, in a predetermined coordinate system (for example, an X1Y1 coordinate system with the center of the platform as the origin, the long side direction of the platform as the X1 axis direction, and the short side direction as the Y1 axis direction) corresponding to the position of the board type controller on the platform, an area (right foot area) to which the center of gravity moves when the user lifts the left foot and lowers the right foot in a stepping motion, and an area (left foot area) to which the center of gravity moves when the user lifts the right foot and lowers the left foot in a stepping motion are set. Here, F indicates the number of frames (the number of times the game processing cycle has been repeated, and the game processing time=1 frame time) from when the stepping flag is turned on to the current time, and indicates the number of frames from when the user's previous stepping on the board type controller is detected to the current time (i.e., when the current stepping is detected). Here, the threshold value used in the above step is a value for counting the stepping of the user by detecting that the load that rose once by the user stepping on the board type controller has dropped, and may be a preset value or a value set by the user's operation. Here, the threshold value used in the above steps is a value for detecting that the user has stepped on the board type controller, and may be a preset value or a value set by the user's operation. In addition, in the above-mentioned stepping determination, it is determined whether the user has stepped on the board type controller with his / her left foot or right foot, but such a distinction between left and right may not be made.

[0105] Alternatively, the user may move the cursor to the display position of "activity amount" and press the switch / determine button, and then each time the up button or down button is pressed, "activity amount" and "calories burned" are alternately displayed (and the unit of measure is alternately displayed in conjunction with this), so that the user can select the input item. Also, based on the calorie consumption per game, the number of times to participate in a game (for example, the number of times to participate in the game) required to consume the target calorie consumption amount or the general calorie intake may be calculated, and the number of times may be displayed on the UI section of the terminal device. The distance traveled when not participating in a game may be measured by the terminal device or a pedometer, and the number of games to be participated in and the distance traveled, etc., to consume the target calorie amount may be calculated based on the distance traveled and the distance traveled during the game. In the above-mentioned embodiment and modified example, the control section may calculate the remaining energy (calories) (=total calories-calories consumed) that can be consumed in a day based on the total calories required in a day and the energy (calories) of food actually consumed by the user, and provide the user with information indicating the remaining calories that can be consumed. The exercise evaluation unit may compare the accumulated exercise calories with the total calories ingested by the user within a predetermined period (e.g., one day), or may compare the accumulated exercise calories with the carbohydrate calories ingested by the user within a predetermined period. Method of Determination Using Infrared Sensor In the current mainstream HMD, an infrared sensor is used to track the user's position information so that the user can move freely around the virtual space.

[0106] A system described in any one of appendices to 7, wherein in the sensing step, the first processor identifies a specific evaluation value regarding the first user's body movements based on the point cloud data of the first user, and at least one of the first processor, the second processor, or a processor of a computer capable of communicating with the first user's terminal and the second user's terminal identifies a third user who will train together with the first user based on the evaluation value of the first user, and in the presenting step to the second user, the second processor receives information based on the point cloud data of the first user and point cloud data sensed from the third user's body movements, thereby presenting an image to the second user that includes the first avatar and a second avatar corresponding to the body movements of the third user. The item "Sensing results" refers to the results (point cloud data) of the trainer's body movements during training measured by a terminal device using a measurement unit, or an image in which the point cloud data is placed in a virtual space and rendered based on the settings of a virtual camera (the image includes an avatar that represents the trainee's body movements as a collection of point cloud data, and auxiliary lines drawn according to the trainee's body movements (the type of body movements presented to the user, such as lateral raises)).

[0107] In this case, as a method of specifying the physical fitness level, when the terminal device or server senses the user's body movement, the user's physical fitness level may be specified based on a learned model or the like, or a user (trainer) who is in a position to instruct the user (trainee) receiving training may specify the trainee's physical fitness level based on the results of sensing. For example, the user may set the way of moving the body on the terminal device, and while drawing auxiliary lines or the like according to the set way of moving the body, notify the user of the information on how to move the body (guide the user on how to move the body), and sense the way the user moves the body with a measurement unit or the like. In this case, if it is determined that accurate sensing is not being performed by analyzing the acquired biological information, the accuracy of sensing can be further improved by changing the measurement parameters, such as changing the measurement wavelength band of the sensor unit or changing the measurement time.

[0108] This allows the user to watch the race between his / her robot device and another user's virtual robot on a race mat in his / her home. On the other hand, walking actions such as jogging are less dangerous than mountain climbing or hiking because the user can stop walking at any time when he / she feels that his / her physical strength is at its limit. However, if the user can grasp the amount of exercise corresponding to his / her physical strength beforehand and perform walking actions, jogging will not damage his / her physical condition, so it is also a good idea for health management. The "posture" may be determined so that if "jogging" is input as the exercise type to be input, the character is displayed in a running posture, and if "walking" is input, the character is displayed in a walking posture. In general, when continuing to walk such as mountain climbing, hiking, or jogging, which consumes physical strength, it is important to make a plan according to the physical strength of the walking performer. After controlling the steps, the processor may cause the HMD to display a screen that prompts the user to place the controller between the user's legs.

[0109] The movement of the virtual person or the like may be any of upward, downward, leftward, rightward, etc., and may be in a direction along the direction of the movement of the upper body, or may be in another direction. In this respect, in this embodiment, when the movement state of the robot becomes a predetermined state (high-speed rotational movement, etc.), the displayed objects other than the enemy robot ERB that the user is gazing at become blurred images. Here, the orthogonal transformation means a coordinate transformation such that the distance between points does not change due to the transformation, such as a parallel movement, a rotational movement, an inversion (mirror image) movement, etc. The movement may include a speed, an acceleration, or a position change (rotation, movement from one place to another place, etc.). The application control program according to any one of claims 1 to 5, further causing the control unit to function as a means for moving a moving object that is a target of a second action in an image of the virtual space output to the wearable display, the means for performing the first action stops the movement of the moving object based on an operation of the display as the first action, and the means for outputting the image of the virtual space reflecting the first action stops the movement of the moving object in the image output to the wearable display. Furthermore, in the "flight mode", when the left foot position and the right foot position exceed the front boundary and the position of the tracking sensor attached to the head or the tracking sensor attached to the upper body other than the head moves to a rear position relative to the initial position at the start of the game, that is, when the head or the upper body other than the head is tilted in a direction opposite to the direction in which the legs are extended, the speed and distance of movement of the seat object and the virtual person may be increased to reflect the action. The application control program according to any one of claims 1 to 5, wherein the means for outputting an image of the virtual space in which the first action is reflected displays a predetermined object included in the overlapping part when the determined display range and the attention range according to the posture of the first user overlap, and outputs the predetermined object in a non-displayed state to the wearable display when there is a predetermined object not included in the overlapping part.

[0110] Conversely, in the relationship between a pet and its master, when the face image of the pet is selected, the face images corresponding to the master and his / her family smile or turn their gaze toward the pet, and the game device can reflect the emotions, consciousness, and relationship of the real world in the virtual world. Next, an explanatory diagram showing an example of a state in which a guide (an electronic pet walking on a sidewalk) is out of the range of the camera and is no longer displayed on the display, and the guide (an electronic pet that is out of the range of the camera and is no longer displayed on the display) cannot be displayed on the display at all. The game device may select the next face image to be used from the game execution history by the user up to the present time based on the face image as well as the attributes of the subject of the face image, such as age, sex, friendship (interactions with family, friends, workplaces, schools, local areas, etc.), and ownership of the subject when the subject is a living creature such as a pet. For example, the following reference document discloses a technology that makes it easier to recognize the presence of a virtual object by transitioning the internal state of the virtual object through a predetermined action by a predetermined part of the user in a mixed reality presentation device such as a game in which a virtual pet is raised. The information processing device may further include a switching unit (and a purchasing unit) that allows the user to select a pet or character to interact with from a plurality of types of pets or characters, either free of charge or for a fee, and causes the selected pet or character to appear in the virtual reality space.

[0111] In the above embodiment, the goggles worn by the cat are closed to prevent the cat from finding the mouse. In another example, when the first elapsed time reaches the first predetermined time, a character such as a dog may appear and block the cat's view, or the cat may be forced to move backwards and close the curtain. For this reason, a predetermined range of depth values ​​(e.g., 0.9-1.0) is stored in each pixel of the area where the dog object is displayed (the display area of ​​the silhouette image), and a predetermined depth value (e.g., 0) is stored in the area where the dog object is not displayed. For example, in order to obtain face images of various animals such as mammals such as dogs, cats, and horses, birds, fish, reptiles, amphibians, and insects, the face images may be collected by executing the face image acquisition process described in the first embodiment. When the dog is not interested in the player at all, the dog ignores the player's touch on the touch panel (normal mode), and when the dog is very interested in the player, the dog licks the point touched by the player (licking mode to be described later). For example, a color image of a dog object (which may or may not be a stereoscopic image) may be displayed in one area of ​​the screen, and a silhouette image of the dog object may be displayed in the other area of ​​the screen.

[0112] In the case of a cat object capture event, the state detection processing unit determines that the player's position in the real space indicated by the player state (specifically, the position of the center of gravity or the position of a predetermined part such as the hand or head) has reached the end position on the moving path R, and also determines that the player's posture is crouched and that the positions of both hands in the real space have a predetermined positional relationship with the real space position of the object to be captured (i.e., when contact is determined by contact determination), and determines that the player state is a cat object capture event occurrence state. On the other hand, as with the player falling state, the state detection processing unit may change the above-mentioned determination criteria and determination conditions according to the game environment such as the player's level (e.g., whether the player is an advanced player or a beginner) and the player's attributes (e.g., gender, age, height), or according to the progress of the game such as when the player's moving speed is above a certain level, when the time limit is approaching, or when it is taking a long time to capture the cat object. For example, when the pseudo-moving object is an animal object such as a cat, the movement / motion processing unit performs calculations of movement information and motion information so that the pseudo-moving object moves at a position different from the position where it is placed in the real space, or moves around a different area, and returns to the position where the pseudo-moving object is placed in the real space at a predetermined timing, and becomes in a state similar to the state where it is placed in the real space. For example, in order to acquire face images of various animals such as mammals such as dogs, cats, and horses, birds, fish, reptiles, amphibians, and insects, the face image acquisition process described in the first embodiment above may be executed to collect the face images. The target object may be a human being, a dog, a cat, or other living thing, or may be an inanimate object such as a robot.

[0113] An information processing method executed by a computer for providing a virtual space to a user via a head-mounted display, comprising: a step of controlling a virtual camera that defines a field of view image from a first character object in a virtual space including a first character object associated with a first user and a second character object associated with a second user; and a position control step of setting a position of the virtual camera to a height based on the first character object and setting the second character object according to the size of the second character object when the virtual space is set to a normal mode in which the first user and the second user share a virtual experience in the virtual space via the first character object and the second character object, respectively, before a specified content is played, and controlling the height of the virtual camera and the eye height of the second character object to be within a certain range of positional relationship in the vertical direction in the virtual space when the virtual space is set to a viewing mode in which the specified content is being played.

[0114] An information processing method executed by a computer for providing a virtual space to a first user via a first head mounted display (HMD device), comprising: generating virtual space data defining a virtual space including a first player character associated with the first user, a second player character associated with a second user, and a virtual camera defining a field of view image to be provided to the first head mounted display; and setting a transmittance of a second player character included in the field of view image, the second player character including a first part (e.g., head and hands) whose movement in the virtual space is controlled in accordance with movement of the second user, and a second part (e.g., torso and arms) displayed in association with the first part, and in the setting step, the transmittance of at least a part of the second part is set to be greater than the transmittance of the first part.

[0115] Here, it is described that movies and games can be enjoyed together in a virtual space, and that the layout of a house can be reproduced three-dimensionally in the virtual space. However, these are movies and games enjoyed together in a pre-set virtual space, and are not intended to reproduce the experience of enjoying a game, especially a competitive game, together at one's own or a friend's home. Although it is possible to reproduce a situation of returning home by setting the layout of the house oneself, it is not possible to reproduce an experience with a friend in the virtual space. The virtual user may be displayed in the VR space as an avatar image including the whole body, upper body, or face (part or all of one's appearance is visualized as an avatar image), or may be displayed in the VR space only as an image of a virtual hand or controller (one's appearance is not visualized as an avatar image). The online game image may further include various other objects, such as game characters such as NPCs (non-player characters) that appear in the online game, scenery such as buildings and trees, furniture in the room and interior of the walls, items lying on the ground, parts of the body (hands and feet) of the avatar (oneself) operated by the user, and objects (guns and swords) held by the avatar in its hands. Furthermore, when the detected biometric information such as heart rate and blood pressure are normal, they are not used to display the character, and only when the biometric information is abnormal, the biometric information may be used as the subject of the character display, thereby executing a character display that has the meaning of a warning or caution for the user or others. [Adjustment of Bone Model] The information processing device of this embodiment may further control the display position of the character's hands and feet as viewed from the character's head to match the position (angle from the viewpoint) when the user views the user's own hands and feet in real space, by changing the arm length relative to the height of the bone model of the virtual character used.

[0116] The virtual mirror control and virtual system encompasses a broad combination of interfaces, including playback of recorded footage (automatic, manual), progressive effects (color editing of items, augmented reality features, game features, background changes, 3D effects, lighting effects, etc.), screen modes (orientation, full size or split screen), sharing techniques and methods for sharing experiences with friends and experts in real time and on the spot, mobile user applications for controlling display and sharing (e.g., sharing personal footage and images captured from the virtual mirror or the user's mobile device), retail or enterprise remote for general management (providing real-time retail information about the user to advanced control and analytical functions), and modules for teaching the user how to operate the mirror (animation, video, audio, visual guidance, etc.). In one embodiment, the mirror can be adapted to exhibit pre-determined characteristics based on information about the user's familiarity with the system, for example, based on whether the user is a first-time user, based on the number of user footage in the system, based on when the user last requested the system to enable instructions, etc., provide relatively simple functionality such as voice assistance, provide remote expert assistance, etc. The periscope disclosed in this patent document synthesizes a corresponding virtual image with the captured image of the viewing object, and displays it, making it possible to provide the user with a highly realistic synthetic image with accurate scale of buildings and their positional relationship on the terrain, without forcing the user to alternate between imagining the actual local scenery and the virtual image of ruins, etc. In this way, when a first-person viewpoint image is displayed on the monitor, if a user wearing an HMD turns his head downward, the inside of the avatar object is not drawn, but the mirror object placed in the virtual space displays the entire appearance of the user's avatar object. Here, the data of the three-dimensional shape of the heart is generated using CG from images such as MRI (Magnetic Resonance Imaging), and the geometric consistency with the position of the endoscope is acquired in advance, and the three-dimensional shape of the CG and the virtual shape are placed in a state of positional consistency in the same three-dimensional space.

[0117] An information processing method executed by a computer in an information processing system including a first user terminal having a first head-mounted device worn on the head of a first user, and a second user terminal having a second head-mounted device worn on the head of a second user, the information processing method including: determining a relative positional relationship between a first avatar associated with the first user and a second avatar associated with the second user; acquiring first virtual space data defining a first virtual space including the first avatar and the second avatar; acquiring second virtual space data defining a second virtual space including the first avatar and the second avatar; and calculating a position of the first head-mounted device based on a movement of the first head-mounted device and the first virtual space data. The method includes a step of updating first field of view image data indicating a first field of view image displayed on the front device, and a step of updating second field of view image data indicating a second field of view image displayed on the second head-mounted device based on movement of the second head-mounted device and second virtual space data, wherein in the first virtual space, a relative positional relationship between the first avatar and the second avatar is maintained, and the distance between a center position of the first virtual space and the first avatar is smaller than the distance between a center position of the first virtual space and the second avatar, and in the second virtual space, a relative positional relationship between the first avatar and the second avatar is maintained, and the distance between a center position of the second virtual space and the second avatar is smaller than the distance between a center position of the second virtual space and the first avatar.

[0118] The virtual environment may be used to simulate a real environment in reality, for example, the virtual environment may include sky, land, or sea, the land may include environmental elements such as a desert or a city, the user may control virtual objects to move in the virtual environment, the virtual objects may be virtual avatars representing the user in the virtual environment, the virtual avatars may take any form, such as a human or animal, and the application is not limited thereto.

[0119] A method for transforming a virtual object located in a three-dimensional space, the method being performed under control of an augmented reality system comprising computer hardware and a user input device, the AR system being configured to enable user interaction with a virtual object within the user's kinematic field of view, the FOR including a portion of the user's surrounding environment perceivable by the user through the AR system, the virtual object being presented via the AR system for display to the user, the method including: determining a group of virtual objects in the user's FOR; receiving a selection of a target virtual object within the group of virtual objects in the user's FOR; calculating a distance to the target virtual object; determining a multiplier based at least in part on the distance to the target virtual object; receiving a first movement of the user input device; calculating a second movement of the target virtual object, the second movement based at least in part on the first movement and the multiplier; and moving the target virtual object by an amount based at least in part on the second movement.

[0120] Similarly, the hardware for the video playback apparatus may be a conventional computing device operating under suitable software instructions, such as a playback processor (e.g., a CPU operating under suitable software instructions) configured to acquire a first video recording of a video game play session including a sequence of video images, configured (e.g., by suitable software instructions) to acquire a second video recording generated by encoding a sequence of depth buffer values, and configured to acquire a sequence of in-game virtual camera positions associated with at least one of the acquired video recordings, and configured to provide the in-game virtual camera positions to an event analyzer for statistical analysis. the analysis processor (e.g., a CPU operating under suitable software instructions) configured (e.g., by suitable software instructions) to obtain data representing statistically significant in-game events and in-game event locations; a position calculation processor (e.g., a CPU operating under suitable software instructions) configured to calculate a location in a current video image of the first video recording that corresponds to the in-game event location responsive to an associated in-game virtual camera position and a decoded depth buffer value; and an augmentation processor (e.g., a CPU and / or GPU operating under suitable software instructions) configured to augment the current video image with a graphical representation of the statistically significant in-game event responsive to the calculated location.A mapping method may thus include the steps of generating at least a portion of a first map including a graphical representation of some or all of the mapping points of at least a first map data set for display with a first video recording; generating at least a portion of a second map including a graphical representation of some or all of the mapping points of at least a second map data set associated with a different video recording of a different sequence of video images output by the same video game title and sharing the same in-game coordinate system; displaying at least a portion of the first map during playback of the first video recording, the displayed portion including at least a current virtual camera position associated with the displayed video images; and displaying at least a portion of the second map during playback of the first video recording if a respective portion of the second map is within a predetermined range of the current virtual camera position of the in-game coordinate system. In other words, the CPU may acquire data for outputting information, move the output source and / or the virtual observation point in a virtual space in which an output source that outputs the information and a virtual observation point that virtually observes the information virtually output from the output source are respectively set, and use a value that limits the speed of the output source and / or the magnitude of the speed of the virtual observation point to add an effect using the limited value to the information that is virtually observed at the virtual observation point when the information is output from the output source, thereby generating modified information, and outputting this modified information.

[0121] The virtual environment may be used to simulate a real environment in reality, for example, the virtual environment may include sky, land, or sea, and the land may include environmental elements such as deserts and cities, and the user may control virtual objects to move in the virtual environment, which may be virtual avatars representing the user in the virtual environment, and the virtual avatars may be any form, such as humans or animals, and the application is not limited thereto. The use of multiple labelers (children and caregivers) providing several signals can reduce errors, and a module measuring the rationality of the labels (using the agreement level of the human labelers in conjunction with machine learning algorithms) can decide whether further expert labelers should determine the truth value of the representation, the human input should be taken as the truth value, or the data should be discarded. In both cases, asking humans to manually correct the labels frame by frame is infeasible, and therefore these embodiments may take an active learning perspective, which relies on various heuristics to select the data to be used in these queries. In the above embodiment, the angle of view of the virtual camera in the initial field of view is an angle of view in which the image field of view is narrower than the human field of view, but the angle of view of the virtual camera in the initial field of view may be an angle of view in which the image field of view is the same width as the human field of view. Specifically, the "predetermined criteria" may include, for example, some or all of whether the motion is executable by a human, whether the motion is executable by a non-human monster or the like, and whether the motion is executable by an artificial object other than a virtual living being.

[0122] For example, an icon representing an avatar may be displayed at the current position of the avatar on a plan view of a golf course as shown in the figure. Also, for example, when a user other than the Oni (child) finds the avatar of the Oni user, he or she can press the lower part of the touchpad to make his or her avatar run away from the avatar of the other user. For example, when a user who is the Oni finds the avatar of another user, he or she can press the upper part of the touchpad to make his or her avatar run toward the avatar of the other user.

[0123] The use of multiple labelers (children and caregivers) giving several signals can reduce errors, and a module measuring the rationality of the labels (using the agreement level of the human labelers in conjunction with a machine-learned algorithm) can determine whether a further expert labeler should determine the truth value of the representation, whether the human input should be taken as the truth value, or whether the data should be discarded. When the child object moves from a position farther away from the user than a predetermined distance as shown in the figure to a position less than a predetermined distance as shown in the figure, that is, when the child object approaches and comes within a range where the user can touch it, the child object is displayed on the screen of the flat image display device. For example, when the child object and the furniture object are displayed on the screen of the stereoscopic image display device, when only the child object is operable (when selected), the child object may be displayed in red on the screen of the flat image display device, and the other objects may be displayed in gray. Therefore, the image displayed in the flat image display area (the image of the virtual space including the child object) is an image of approximately the same size as the image displayed on the stereoscopic image display device (the image of the virtual space including the child object and the furniture object), and is an image of approximately the same range of the virtual space. In addition, the item may be pre-existing at a specified position on the screen of the stereoscopic image display device (a specified position in the virtual space), and the user may touch the item image displayed on the flat image display device to hand it over to the child object.

[0124] In addition, at least one of the system software, portal software, and game application may not be pre-stored at the time of sale of the information processing device, but may be provided by downloading via a network, and may be updated to the latest version by a network update.

[0125] The resolution of the eye camera may be set to high resolution when the user is wearing the wearable device so that a high resolution photograph of the user's entire face is available for use in generating a model of the user's face. In addition, in a ride attraction for children, a composite image (composite video) of a child riding on a ride cabinet may be generated and printed as a photograph to be taken home or distributed. In addition, for example, the camera image may be an image (e.g., a landscape photo) stored in an external data storage memory of the game device. In addition, the map image is image data representing the area where the panoramic photograph was taken, and may be a photograph or CG. The "predetermined texture" may be a monochromatic image (e.g., a black image), or may be a photograph or CG image representing the ground, floor, or surface of the earth.

[0126] The online game image may further include various other objects such as game characters such as NPCs (non-player characters) appearing in the online game, scenery such as buildings and trees, furniture in a room and interior decoration of walls, items lying on the ground, a part of the body (hands and feet) of the avatar (the user himself) operated by the user, and an object (gun or sword) held by the avatar. In addition, when a selected virtual space image related to a scenery seen when driving on a road along a mountain or a sea is displayed, if a turning virtual space image related to a seaside scenery is displayed while a selected virtual space image related to a mountain scenery is displayed, the user feels uncomfortable. In addition, when the user terminal captures an image of the surrounding scenery and displays it on the display unit, a virtual object image indicated by the image data may be displayed on the display unit based on object position information received from the distribution device. In addition, since the operator basically does not perform a shot operation when this camera control function is activated, a sound corresponding to the displayed scenery (view) may be output according to the operation of the controller. In the above embodiment, an example was given of a virtual character having a human form singing and dancing to music, but the subject object may be an animal, a plant, a landscape, a vehicle, or the like.

[0127] In the above embodiment, the left and right positions of the left and right fist objects in the virtual space are controlled in response to the operation using the left and right controllers, and the vibration changes temporarily in response to the operation. However, the left and right fist objects may be configured to be controllable in the vertical and / or front-back positions in the virtual space. For example, in the above game example, the arms of the first player object are stretched to control the movement during the movement, but this embodiment may be applied to a game in which the other five bodies (e.g., legs) of the player object are stretched, or a game in which an object held by the player object (whip object, bellows object, etc.) is stretched. In the above embodiment, the virtual microphone is associated with the player object, and the virtual sound source is associated with the non-player object, but this embodiment is not limited to this. For example, the virtual sound source may be associated with the player object (i.e., the position and speed of the virtual sound source may be controllable by the user). In this case, a door in real space may be fitted with a key that can be actually controlled to be locked and unlocked from the server device, and the server device may control the door to be locked when a player belonging to a group that is not allowed to pass through that door approaches that door. In this case, the vertical positions of the left fist object and / or right fist object in virtual space may be controllable in response to the vertical movement and / or pitch orientation of the left controller and / or right controller in real space, and the vibration may temporarily change in response to that operation.

[0128] In the shared mode, when the surgeon specifies a position on the virtual object with a finger, the specified position information is acquired by the specified position information acquisition unit, and the display control unit of the participant's HMD may display an image of an arrow representing the specified position on the virtual object displayed on the participant's display based on the position information acquired by the specified position information acquisition unit. Here, for example, when the surgeon stands in front of the virtual object, the participant stands opposite the surgeon, the participant stands to the right of the surgeon, and the participant stands to the left of the surgeon, the participant observes the back side of the virtual object of the head, the participant observes the left side of the virtual object of the head, and the participant observes the left side of the virtual object of the head. In addition, in this embodiment, the display of the virtual object of the participant's HMD is changed in conjunction with the change in the display of the virtual object of the surgeon's HMD as described above, but the display change of the virtual object of each participant's HMD may be performed independently in each HMD without performing such a linked display. Although the operator's position and the hand position for virtual object manipulation are measured, virtual object manipulation is possible, but the stereo image observed by the participant is an image of the participant with the background removed and an image of the VR space consisting of virtual objects, and does not contain information about the participant's real space. Next, when the surgeon instructs to switch to normal mode using the switching controller while in the shared mode, the mode is switched to normal mode, and the virtual object based on the positional relationship between each participant and the marker, that is, the virtual object oriented according to the position of each participant, is displayed on the display of the participant's HMD.

[0129] Furthermore, for example, when a first finger determined to be facing ventrally and a second finger determined to be facing dorsally are both in contact with a target virtual object, and the distance between the coordinate value (x1, y1, z1) of the fingertip in the XYZ coordinate system of the virtual space of the fingertip and the coordinate value (x2, y2, z2) of the fingertip in the XYZ coordinate system of the virtual space of the fingertip is below a predetermined threshold value (at this time, the part of the target virtual object corresponding to the finger facing dorsally is deformed to be located behind the finger, but the position comparison in the Z-axis direction with the finger facing ventrally is performed using the coordinate value before deformation, so the finger is in a semi-transparent composite and drawn state), the information processing device may determine that the target virtual object is virtually sandwiched between the fingers at the position of the fingers (a predetermined range including (x1, y1) and (x2, y2), for example, the range of the smallest circle including these), and execute a predetermined process. Furthermore, in one example of this embodiment, when it is determined that the coordinates of the fingertip of the dorsal finger satisfy the above contact condition, the information processing device compares the Z-axis value (defined as Zj(Xf, Yf)) of a target virtual object located at the position coordinate (Xf, Yf) of the finger as seen from the user's side with the Z-axis value of the finger's position coordinate. If Zth>Zf-Zj(Xf, Yf)>0, the information processing device deforms a part of the target virtual object so that a part of the target virtual object corresponding to the position of the finger is located farther away from the position of the user's finger. If Zf-Zj(Xf, Yf)≧Zth, the position of the entire target virtual object is translated in the Z-axis direction to set the part of the target virtual object corresponding to the finger farther away from the finger. That is, CPU 31 may acquire data for outputting information, move the output source and / or the virtual observation point in a virtual space in which an output source that outputs this information and a virtual observation point that virtually observes the information virtually output from this output source are respectively set, and use a value that limits the speed of the output source and / or the magnitude of the speed of the virtual observation point to add an effect using the limited value to the information that is virtually observed at the virtual observation point when the information is output from the output source, thereby generating modified information, and outputting this modified information.In addition, in the second operation mode, when the player moves the position pointed to by the controller (position of the aiming cursor) while pressing the B button (operation button) of the controller, when the position of the aiming cursor enters the peripheral area of ​​the game image (for example, or the peripheral area outside the displayed image), or when the B button is pressed while the aiming cursor is in the peripheral area, the direction of the virtual camera is changed to a direction such that the position of the virtual game space corresponding to the position of the aiming cursor is displayed on the display screen of the monitor (at this time, the viewpoint position of the virtual camera does not need to be changed). The system will preferably be configured to recognize that the user is holding an item and to understand the type of interaction the user is having with the item.

[0130] Here, as described above, the exercise support device according to the present embodiment is configured by a single device, display glasses, and the display unit has a transparent display surface such as a transparent display panel or transparent glass, so that the user can see the surrounding scenery through the display unit by a simple wearing method of wearing it on the head, and a virtual runner is displayed on the display surface. For example, if the height of the photographer of the shared image and the height of the user who selected the shared image are close, even if the viewpoint of the user is directly moved to the shooting location, the scenery seen after the movement is considered to be closer to the photographer than between users with a large height difference such as a child and an adult, so the viewpoint may be directly moved to the shooting location depending on the combination of the photographer and the user who wants to move. For example, in the above description, the shooting space is a real space, and the shooting image is obtained by shooting a real scenery, but this is not limited to this, and the shooting image may be an image showing the virtual space as seen from a shooting position arranged in the virtual space. The display image in which the user moves in the virtual space may be an image that moves the scenery in the virtual space, or an image with a video effect that evokes movement. In the above description, the captured image is an image that includes scenery within a certain shooting range. However, a panoramic image obtained by capturing a wider range using an omnidirectional camera, for example, may also be used as the captured image.

[0131] In this example of the present embodiment, objects in the real space that are within the reach of the user's hand are reflected as virtual objects in the virtual space, while objects in the real space that are out of the reach of the user's hand are moved or omitted from being displayed as virtual objects, allowing for a highly flexible display. The biokinetic information display device described in the system configuration of the figure may be a device that displays a character image showing the situation of a certain person (a user of the biokinetic information transmission device who is a stranger to the user of the biokinetic information display device) on the display unit based on information received from the server device. The biokinetic information display device described in the system configuration of the figure may be a device that displays a character image showing the situation of a certain person (a user of the biokinetic information transmission device who is a stranger to the user of the biokinetic information display device) on the display unit based on information received from the server device. In other words, when the player is located 1 meter away from the position of the input unit, the hand area of ​​the reflected light image will be brighter (the amount of light received will be greater) than when the player is located 2 meters away from the position. The differences in this figure from the flowchart shown in the figure are the generation of a three-dimensional virtual screen of the steps, the projection of a selected image of the steps onto the three-dimensional virtual screen, and the placement of a virtual camera and a virtual water surface within the three-dimensional virtual screen of the steps.

[0132] After a certain period of use, the patient and his / her caregiver may review the activities stored from the patient's perspective (with reference to the interactions of others with each other and towards the patient) and the patient's reactions to those situations in the form of behavioral and physiological data (e.g., eye contact made, expressions imitated, etc.). The other person detection unit refers to the captured image output from the imaging device, and detects when a person different from the user wearing the HMD on his / her head appears in the captured image. For example, the other person detection unit detects that a person different from the user has appeared in a position near the user when the captured image changes from a state in which no person different from the user is captured in the captured image to a state in which a person different from the user is captured in the captured image. When the character on the screen is clicked, the other person information is displayed in a separate window, such as the "change in body shape", goal achievement rate, and exercise implementation status of the other person. The other person detection unit may detect a person captured in the captured image using a known contour detection technique.

[0133] In the above first and second embodiments, an example was described in which an incentive is given to a virtual user simply by being in the VR space, but a larger incentive may be given to the virtual user as the user actively participates in activities carried out in the VR space. In the above example, the display mode of the player icon in the map image is different between the part visible from the virtual camera 2 and the part invisible from the virtual camera 2. However, the invisible part (hidden part) may not be displayed (however, in this case, since it is difficult to grasp the position of the player icon when the whole is hidden, it is basically preferable to always display the whole image of the player icon while changing the display mode as described above). Here, the basic processing of virtual reality and augmented reality will be described. Still images and objects of CG animation, which are 3DCG (Three-dimensional computer graphics), are mixed and processed (visible / invisible / transparent) based on real map information of a real coordinate system (latitude, longitude, altitude) and displayed on the background and objects of still images and videos, which are real images taken by a smartphone camera or the like. The virtual image generation unit is basically the same as that in the embodiments, except that, whereas in the embodiments, the output image of the game generated by the game execution unit is fitted into the display unit of the television receiver included in the 3D model, in the embodiments, the video output of a stationary game console delivered from a game video delivery unit is fitted into the display unit of the television receiver included in the 3D model.

[0134] In the above embodiment, basically, a determination is made as to whether an event should be generated depending on the approach of the viewpoint to a preset area or the distance between the viewpoint and the AR character, and the behavior information is registered in the behavior list, but the implementation of this embodiment is not limited to this. For example, a basic volume (basic volume) is set for each sound source, and the game device may calculate the volume according to the distance by multiplying the basic volume by a coefficient (a coefficient that changes in the range of 0 to 1.0 depending on the distance) according to the distance between the sound source and the microphone for volume calculation.

[0135] A video, etc. generated based on movement information received from a first sensor (e.g., an inertial sensor) worn by the performer, position information received from a second sensor (e.g., an optical sensor) worn by the performer, part position information calculated based on the movement information, body position information calculated based on the position information, part position information converted from coordinates in physical space to coordinates in virtual space, body position information converted from coordinates in physical space to coordinates in virtual space, part position information converted into coordinates in virtual space and body position information converted into coordinates in virtual space.The sensor unit may include sensors such as various types of cameras and / or microphones, and at least one processor for processing information obtained by such sensors. A process of drawing the face of an avatar based on the measurement data (coordinates of each of a plurality of parts of the face) of the performer's body acquired by the sensor unit, a process of drawing an avatar at coordinates determined based on the position information of the body calculated by the body position information calculation unit, and a process of drawing a part corresponding to each first sensor at coordinates determined based on the position information of the first sensor calculated by the part position information calculation unit. Furthermore, such generation of moving images can be realized by using various rendering techniques well known to those skilled in the art. Note that, in the above-mentioned route, the above-mentioned route, and the above-mentioned route, the terminal device of each viewer does not display moving images generated by the distribution device, but can display moving images generated (drawn) by the terminal device of the viewer itself (this method is sometimes called "client rendering" since the terminal device (client) of the viewer performs the rendering). Here, we will explain the basic processing of virtual reality and augmented reality. Real-life still images and video backgrounds and objects taken with a smartphone camera or similar device are mixed with 3DCG (three-dimensional computer graphics) still images and CG animation objects based on real map information in the real coordinate system (latitude, longitude, altitude) and displayed in a mixed manner (visible / invisible / transparent).

[0136] Short-term feedback may include visual indications of the quality and completion of a repetition, such as a scoreboard that keeps track of the number of repetitions, a virtual character that mimics or reflects the player's movements, animation of the virtual character that cheers, sings, plays music, and / or otherwise motivates the player to continue, physical changes in the virtual reality environment, such as the sun rising in the virtual reality environment, food growing, ice melting, wind blowing, or construction projects (initiated by villagers supported by the player), etc. Although a game machine as a home game machine has been described as an example of a computer that functions the game software according to this embodiment, the game machine may be a so-called portable game machine, and further may not be a device dedicated to games, but may be a device that can play general music and video storage media, and is not limited to this, and the computer may be, for example, a personal computer, a mobile phone, etc., that is, any device that can function the game software. In the present disclosure, a smartphone is given as an example of a device used in the wearing system, but in the present disclosure, a mobile phone, a PHS, a game machine, a watch, a music player, a car navigation system, a television, a heart rate monitor, a pulse wave monitor, a blood flow meter, or any other device or any combination of these may be used instead of or together with the smartphone. Alternatively, when a person is taking a lesson in some movement using footwear (for example, a walking lesson, a running lesson, a dance lesson, etc.), the generation unit may score whether the footwear is suitable for the lesson based on the movement of the footwear, and generate music according to the score. The generation of music according to the movement of the footwear is realized, for example, by storing sounds to be output in advance in a storage unit, and when the movement of the footwear shows an ideal walking style or an ideal movement in a certain action, selecting a sound to play a beautiful melody and generating audio information.

[0137] The reference virtual plane may be "a plane that a user viewing a stereoscopic display feels as if it exists at a position having a predetermined offset from the display surface of the display device in a three-dimensional virtual space."

[0138] Such feedback may be discrete (i.e., indicative of a specific event) or may change over time (e.g., a score such as a "current interest level" score that indicates to the patient the interest level of his / her conversation partner). For example, the game progression unit may award the user with an item that is pre-associated with the scenario played, or may award an item determined according to the play content of the scenario (e.g., the time required to complete the quest, the intimacy level obtained, whether the good option was selected, etc.). The game progression unit may be configured to increase the intimacy level as the play content of the quest (e.g., the time required, the distance traveled, the number of items obtained, the degree of joy of the character, the rarity of the acquired target object, etc.) improves. Furthermore, the game progression unit may increase the intimacy level between the protagonist and the character based on the completion of the quest. In one embodiment, the parental inspection system is a mobile application that runs on the same phone as the social behavior processing program that performs the computationally intensive emotion recognition. Conversely, the game progression unit may decrease the intimacy level when the user reaches a bad end in the scenario.

[0139] Modification Example Also, in the above-described embodiment, in a state where the target object and the first mark overlap (an example when the positional relationship between the display object and the first designated position satisfies the third condition), when an operation is performed on the first operator provided on the operation unit of the first game device, or in a state where the target object and the second mark overlap (an example when the positional relationship between the display object and the second designated position satisfies the third condition), when an operation is performed on the second operator provided on the operation unit of the second game device, an example of a game in which the target object can be destroyed has been described. However, in a state where the first object (an example of the first display object) and the first mark overlap (an example when the positional relationship between the display object and the first designated position satisfies the third condition), when the operation on the first operator provided on the operation unit of the first game device is continued, or in a state where the first object and the second mark overlap (an example when the positional relationship between the display object and the second designated position satisfies the third condition), when the operation on the second operator provided on the operation unit of the second game device is continued, the first object can be moved so as to follow the movement of the first mark or the second mark, or a game in which the moving target object can be stopped may be performed. More generally, in a UCL color video, N-bit depth data (N is typically 16) can be encoded into M bits (M is typically 8) for each color space descriptor channel. Typically, M < P, where P is the original bit depth of each channel in the host data format, and M is preferably 1 bit, more preferably 2 bits less than P. And in the game system, an operation for designating a direction, such as a cross key, is operated so that a first operation for designating a position in the image is performed, and a second operation for designating a position in the image may be performed by contacting a touch panel display with the first touch pen.The process for identifying facial expressions or facial actions may include the following fragments: face detection and / or reference point face tracking face alignment with standard size and posture, illumination normalization preprocessing techniques histogram of gradient feature extraction in the region of interest for the face classification of facial expressions via a logistic regression classifier trained on the image data, and filtering of the resulting classification. In a method of synthesizing vibration components by rotating two eccentric oscillators in opposite rotation directions, when a high-speed rotation frequency ω1 (high frequency f1) at the operating point and a low-speed rotation frequency ω2 (low frequency f2) at the operating point are presented alternately every 180° in phase, the haptic illusion intensity (II) is proportional to the logarithm of the acceleration / deceleration ratio Δf / of the frequency, which is the rotation speed of the eccentric rotor.

[0140] In addition, in the above embodiment, in the virtual camera interval determination process, the interval between the outer imaging unit (left) and the outer imaging unit (right) in the marker coordinate system is calculated, and then the position and orientation of either the left virtual camera or the right virtual camera is determined from the position and orientation of the other virtual camera calculated based on the marker recognition result based on the interval. However, in other embodiments, the position and orientation of the outer imaging unit (for example, the average position of the position of the outer imaging unit (right) and the position of the outer imaging unit (left), the average orientation of the orientation of the outer imaging unit (right) and the orientation of the outer imaging unit (left)) may be calculated based on the position and orientation of the outer imaging unit (left) calculated based on the marker recognition result for the left real world image, and the position and orientation of the outer imaging unit (right) calculated based on the marker recognition result for the right real world image, and the positions and / or orientations of the left virtual camera and right virtual camera may be determined based on this. In another example, a real world image may be captured by a first virtual camera (hereinafter referred to as a real world drawing camera) and rendered to form a first rendering image, and a virtual object may be captured by a second virtual camera (hereinafter referred to as a virtual world drawing camera) and rendered to form a second rendering image, and the first rendering image and the second rendering image may be composited so that virtual objects existing in the foreground area are given priority over the real world image, and the real world image is given priority over virtual objects existing in the background area. Furthermore, if we assume that there is absolutely no error in the marker recognition accuracy and no error in the attachment accuracy of the outer imaging unit (left) and the outer imaging unit (right) to the game device, the position of the outer imaging unit (right) indicated by the right transformation matrix, which is the marker recognition result of the right real world image, will be a position shifted a certain distance (distance between the imaging units) along the x-axis direction of the outer imaging unit (left) coordinate system (the lateral direction of the game device, the horizontal direction when in use) from the position of the outer imaging unit (left) indicated by the left transformation matrix, which is the marker recognition result of the left real world image, and the orientation of the outer imaging unit (right) indicated by the right transformation matrix and the orientation of the outer imaging unit (left) indicated by the left transformation matrix will be the same as the attachment state of the outer imaging unit (left) and the outer imaging unit (right) on the game device.ARActive is a variable (flag) indicating whether or not a virtual object can be compositely displayed on a real world image. In a situation where a virtual object can be compositely displayed on a real world image, the value is set to true, and in a situation where a virtual object cannot be compositely displayed on a real world image (for example, a situation where a marker cannot be recognized at all), the value is set to false (initial value). For example, after calculating the interval between the outer imaging unit (left) and the outer imaging unit (right) in the virtual camera interval determination process, the positions of the left virtual camera and the right virtual camera may be determined by shifting them in opposite directions by EyeWidth / 2 from positions in the virtual space corresponding to the average position of the position of the outer imaging unit (left) calculated based on the marker recognition result for the left real world image and the position of the outer imaging unit (right) calculated based on the marker recognition result for the right real world image.

[0141] Similarly, the hardware for the video playback apparatus may be a conventional computing device operating under suitable software instructions, such as a playback processor (e.g., a CPU operating under suitable software instructions) configured to acquire a first video recording of a video game play session including a sequence of video images, configured (e.g., by suitable software instructions) to acquire a second video recording generated by encoding a sequence of depth buffer values, and configured to acquire a sequence of in-game virtual camera positions associated with at least one of the acquired video recordings, and configured to provide the in-game virtual camera positions to an event analyzer for statistical analysis. the analysis processor (e.g., a CPU operating under suitable software instructions) configured (e.g., by suitable software instructions) to obtain data representing statistically significant in-game events and in-game event locations; a position calculation processor (e.g., a CPU operating under suitable software instructions) configured to calculate a location in a current video image of the first video recording that corresponds to the in-game event location responsive to an associated in-game virtual camera position and a decoded depth buffer value; and an augmentation processor (e.g., a CPU and / or GPU operating under suitable software instructions) configured to augment the current video image with a graphical representation of the statistically significant in-game event responsive to the calculated location.A mapping method may thus include the steps of generating at least a portion of a first map including a graphical representation of some or all of the mapping points of at least a first map data set for display with a first video recording; generating at least a portion of a second map including a graphical representation of some or all of the mapping points of at least a second map data set associated with a different video recording of a different sequence of video images output by the same video game title and sharing the same in-game coordinate system; displaying at least a portion of the first map during playback of the first video recording, the displayed portion including at least a current virtual camera position associated with the displayed video images; and displaying at least a portion of the second map during playback of the first video recording if a respective portion of the second map is within a predetermined range of the current virtual camera position of the in-game coordinate system.

[0142] On the other hand, the app allows the user to send a link, for example, via email or SMS, to allow a third party cloud to access the real-time mirror reproduction streaming video. For example, if the user is not identified, a code such as a QR code may be displayed on the monitor so that the user can scan it with a mobile device to download the app. When the app is executed, the terminal display of the mobile terminal displays application functions (specifically, icons indicating the application functions) for the mobile terminal to function as a controller. For example, the storage may be paused when there is a notification from the system of the user terminal (power loss, notification from another app, incoming call, etc.). The user may be provided with their own app or portal that allows access to progress data, session data, and appointment schedules.

[0143] In addition, in this embodiment, a chest sensor attached to the user's chest, a wrist analyzer attached to the wrist, and a foot sensor attached to the ankle, shoelaces, soles, etc. are shown as sensor devices equipped with a sensor function, but the present invention is not limited to these, and sensor devices may be attached to any part of the body, such as the waist, upper arm, or neck.

[0144] In the present embodiment, a chest sensor attached to the user's chest, a wrist analyzer attached to the wrist, and a foot sensor attached to the ankle, shoelace, sole, etc., are shown as sensor devices having a sensor function, but the present invention is not limited to these, and sensor devices attached to any part, such as the waist, upper arm, or neck, may be applied. (Device attached to wrist) As shown in the figure, a device (which may be the detection device shown in the figure or a similar device) may be configured to be attached around the user's wrist, arm, ankle, etc. On the other hand, when determining whether to execute an action based on a change in the moving direction of the user's ankle position, for example, when the user moves his or her foot slightly, the action may be executed at a timing unintended by the user, which may cause the user to feel uncomfortable. Alternatively or in addition to this, a tracking sensor attached to a position on the user's body other than near the ankle may be used to determine whether to start executing an action and / or whether to complete the executed action. The game processing device may determine that a kick is a dribble when a kicking action is started from an ankle position lower than a boundary in a game including an action of kicking an object, such as soccer.

[0145] Here, the bone model includes a plurality of nodes and bones connecting pairs of nodes, and the nodes are arranged at positions corresponding to each part of the virtual character, such as the chest, neck NK, head, shoulders, arm joints, wrist joints (arm tips) HD, waist, hip joints, knees, and feet FT. The processor may change the drawing mode of the knee part of the avatar or the auxiliary line associated with the knee part as the knee movement approaches a direction along the auxiliary line in the trainee's body movement. Note that instead of the protective sheet, the player may be required to wear a helmet or protective pads on the shoulders, elbows, knees, etc. The knee and ankle sensors are similarly identified by hemisphere (left or right) and height. Motion sensors may further be attached to both elbows and both knees of the user.

[0146] Here, the objects include, but are not limited to, objects such as a batter, pitcher, catcher, fielder, umpire, ball, mound, home base, first base, second base, third base, fence, and spectator stand, which are necessary for generating the data of the virtual space in the game. The data of the virtual space also includes imaging target information, which is various information related to the imaging target, such as the contents and schedule of the imaged event, information about the performers and speakers, and information indicating the imaging location. The data of the virtual space includes data related to each virtual object constituting the virtual space and data related to a light source that illuminates the virtual space. When generating the data of the virtual space, the processing unit may acquire images of various objects from the storage unit. The data of the virtual space includes data of a panoramic image captured by an omnidirectional camera.

[0147] The user's own team consisting of players owned by the user and the cooperation team are defined by a deck, the deck has a predetermined number of frames for each role constituting the team, and the role of the player is set by linking the player to the frame in the deck, and in the generating step, players may be selected from each user's deck so that the same role as the role set in the user's deck is set for the player in the cooperation team deck and players are linked to all the frames of the roles constituting the cooperation team. For example, the string-like member has a length that does not contact the floor of the structure even when the HMD is attached or detached from the player, the string-like member has an expandable spiral shape, or has a structure for winding up the cable to adjust the length of the string-like member. As another example, the management unit may, for example, lower the movement performance of each character image linked to each user belonging to the team as the team acquires more points, reduce the size of each tracking character image linked to each tracking player belonging to the tracking team, increase the size of each escaping character image linked to each escaping player belonging to the escaping team, or lower the operability of each character image linked to each user belonging to the team. As another example, when multiple character images linked to multiple users move to a specific location in the virtual space, or multiple users move to a specific location in the real space, and at the specific location, a character image linked to a user is overlapped or connected to a character image linked to another user, points may be awarded to the user or the other user. For example, the management unit may lower the movement performance of each character image linked to each user belonging to a team with a larger number of users, may reduce the size of each tracking character image linked to each tracking player belonging to the tracking team, may increase the size of each escaping character image linked to each escaping player belonging to the escaping team, or may lower the operability of each character image linked to each user belonging to the team.

[0148] In order to reproduce the experience of enjoying the game of this embodiment together, it is appropriate to reproduce the actual video game as it actually existed. However, when implementing the video game in a game execution unit provided in a server, it is necessary to create a new program that reproduces the video game, or to create an emulator for a console game in order to execute the program for the video game in the game execution unit provided in the server.

[0149] In the figure, each element described as a functional block performing various processes can be configured in terms of hardware with a circuit block, memory, or other LSI, and in terms of software, it is realized by system software, a game program loaded in memory, or the like. The virtual space image displayed on the display unit of the talent terminal may include a plurality of virtual space images corresponding to views from various positions (virtual viewpoints) in a virtual space corresponding to a certain handshake event venue. Those skilled in the art will understand that in various embodiments described in this disclosure, the aforementioned handheld device is utilized in conjunction with an interactive application displayed on the display to provide various interactive functions. In the figure, each element described as a functional block performing various processes can be configured in terms of hardware with a CPU, main memory, or other LSI (Large Scale Integration) stored in the housing of the stereoscopic image presentation device. It should be noted that although some of the above-mentioned embodiments relate to transmitting a signal from a pad device to an HMD, in various embodiments, the signal is transmitted from the pad device to a game machine.

[0150] The premium / discount coefficient is a coefficient for moving the hand of the simulated experiencer, which is the body, in the direction of a certain area (e.g., a danger area) in the virtual space more or less than the intended direction of the simulated experiencer, and when the inner product of a unit vector facing in the same direction as the movement amount vector Y(n-1) and a unit vector in the direction of looking at a certain area from the body in the virtual space is greater than a predetermined value, the premium / discount coefficient is set to a value greater than 1 to allow the hand to be moved more than the intended direction of the simulated experiencer, and the premium / discount coefficient is set to a value less than 1 to allow the hand to be moved less than the intended direction of the simulated experiencer. Note that in the above embodiment and modified example, before recognition, the marker is displayed on a monitor, the marker displayed on the monitor is imaged and saved, and when recognition is performed, the imaged and saved marker is compared with a newly imaged marker, but by storing the marker in advance, such a saving process can be omitted, and when recognition is performed, the previously stored marker can be compared with a newly imaged marker. In the above embodiment, the terrain object itself is deformed in the step shown in the figure (i.e., each vertex of the terrain object is moved), so that the moving area is displayed as if it is moving, but in other embodiments, the terrain object itself is not deformed, and instead the moving area may be displayed as if it is moving by changing the texture coordinates used in texture mapping on the terrain object. If the game device has a setting in which the viewpoint is a first-person viewpoint during imaging and a third-person viewpoint during normal times, it becomes easy to view and operate the player object by placing the left and right virtual cameras in positions other than the viewpoint of the player object during normal times, and when saving a stereoscopic image, it is possible to save a stereoscopic image with a sense of realism as if it were seen from the viewpoint of the player object by placing the left and right virtual cameras in positions corresponding to the viewpoint of the player object.When performing virtual-reality synthesis processing of playing an actual musical instrument in a virtual space as described above, if semantic segmentation inference processing is constantly performed in the trained semantic segmentation model section on the dual camera image data obtained from the dual cameras attached to the HMD, the inference processing is performed even when the performer is not looking at the instrument or his or her own body, and if an error occurs in the inference result, noise may be mixed into the virtual-reality synthesis image data.

[0151] The present embodiment further relates to a computer-implemented method for generating a visual virtual agent configured to perform an artificial physiological color change response to a user for at least one visual virtual agent state and / or for at least one state of a user, the visual virtual agent being configured to perform a visual response to a user, the at least one visual virtual agent state and / or the state of the user being determined based on user specific parameters including user activity data and / or user physiological and psychological data, the at least one user specific parameter being acquired using at least one sensor, the visual response being dependent on changes in the user activity data and / or the user physiological and psychological data, the visual response at least partially comprising a color change of the visual virtual agent for the at least one determined visual virtual agent state and / or the at least one determined state of the user, the method comprising: acquiring, using at least one sensor, the user activity data and / or the user physiological and psychological data over time; acquiring, from the at least one sensor, the user activity data and / or the user physiological and psychological data over time; acquiring user activity data and / or user physiology with a processor, analyzing the user activity data and / or the user physiology with a processor, determining a visual virtual agent state based on the user acquired activity data and / or the user acquired physiology with a processor and / or determining a user's current state based on the received acquired activity data and / or the user acquired physiology with a processor, selecting at least one sensor for generating a color change response based on the user determined current state and / or based on the determined visual virtual agent state, depicting with a graphics processing unit the visual virtual agent in a first color for the determined visual virtual agent state and / or the user determined current state, displaying on a display device, on at least one display, the visual virtual agent in a first color for the determined visual virtual agent state and / or the user determined current state, using the selected at least one sensor,The method includes the steps of: acquiring (in real time) by a processor user activity data and / or user physiological and psychological data for the determined visual virtual agent state and / or the user's current state; analyzing (in real time) by a processor the user acquired activity data and / or the user's physiological data from the selected at least one sensor; determining by a processor a change in the user acquired activity data and / or the user's acquired physiological data; rendering by a graphics processing unit the visual virtual agent in a second color in response to the determined change in the user acquired activity data and / or the user's acquired physiological data; and displaying on a display device the visual virtual agent in the second color for the determined visual virtual agent state and / or the determined current state of the user;

[0152] In the present embodiment, a chest sensor attached to the chest of the user, a wrist analyzer attached to the wrist, and a foot sensor attached to the ankle, shoelace, sole, etc., are shown as sensor devices having a sensor function, but the present invention is not limited thereto, and sensor devices attached to any part, such as the waist, upper arm, or neck, may be applied. In addition, in the case where the tracked character image is an image representing a character originally wearing shoes, when an item (shoes) is acquired, the image of the original shoe part of the tracked character image may be changed. In addition, in the "kick mode", the movement of the upper body may be reflected in the action, such as by increasing the moving distance of the shoe object according to the movement of the upper body. For example, when the lower leg is long, the moving distance of the shoe object may be corrected to be longer than when the lower leg is short. Conversely, when the lower leg is long, the moving distance of the shoe object may be corrected to be shorter than when the lower leg is short.

[0153] In the present embodiment, the image device stores a three-dimensional image representing a three-dimensional object in a virtual space as an original image in an internal storage device in advance, and generates a display image from the three-dimensional image by processing to make the image look realistic and natural to the user so as to give the illusion that the three-dimensional object in the virtual space exists there. However, as a modified example, the display image may be a three-dimensional image consisting of two images with parallax between the left and right eyes. As a result, even though the outer surface of the upper housing faces the front of the player, the game image displayed there is an image taken from above the virtual space, that is, an image that should be displayed when the outer surface of the upper housing is facing downward. Specifically, when driving a cart in a very narrow and poorly visible passage such as in a warehouse, a head-mounted display worn by the driver may display a plan view of the passage from directly above, as well as the driver's own cart, other pedestrians, carts, etc. moving there. Therefore, the virtual space construction unit may estimate what the subject is based on the position and arrangement of the subject part detected from the distance image, and arrange voxels representing the subject based on the estimation result. The vergence-divergence movement of the two eyes relative to each other (i.e. the rolling of the pupils towards or away from each other to converge the gaze of the eyes and fixate them on an object) is closely coupled to the focusing (or "accommodation") of the eye's lenses.

[0154] Modifications In the above-described embodiment, an example was given of a game in which a target object can be destroyed when an operation is performed on a first controller provided on an operation unit of a first game device in a state in which the target object and the first mark overlap (an example of a case in which the positional relationship between a displayed object and the first designated position satisfies the third condition) or when an operation is performed on a second controller provided on an operation unit of a second game device in a state in which the target object and the second mark overlap (an example of a case in which the positional relationship between a displayed object and the second designated position satisfies the third condition). However, it is also possible to play a game in which a target object can be destroyed when an operation is performed on a first controller provided on an operation unit of a second game device in a state in which the target object and the second mark overlap (an example of a case in which the positional relationship between a displayed object and the second designated position satisfies the third condition). A game may be played in which, when an operation is continued with a first controller provided on an operation unit of the first game device while the first object and the second mark overlap (an example of a case where the positional relationship between the displayed object and the second designated position satisfies the third condition), or when an operation is continued with a second controller provided on an operation unit of the second game device while the first object and the second mark overlap (an example of a case where the positional relationship between the displayed object and the second designated position satisfies the third condition), the first object can be moved to follow the movement of the first mark or the second mark, or a moving target object can be stopped. The game execution means (e.g., the game execution unit in the figure) comprises first tracking means (e.g., the head tracking unit in the figure) that tracks the position of the head based on position information of the player's head obtained from a first sensor (e.g., the HMD in the figure) worn on the player's head, second tracking means (e.g., the arm tracking unit in the figure) that tracks the position of the hand based on position information of the player's hand obtained from a second sensor (e.g., the arm sensor in the figure) worn on the player's arm, synchronization means (e.g., the synchronization unit in the figure) that synchronizes first tracking information obtained by the first tracking means tracking the head position and second tracking information obtained by the second arm tracking means tracking the hand position to obtain position information of the head and hand at the same timing, and game control means (e.g., the game control unit in the figure) that performs control to display images of objects aligned with the positions of the head and hands obtained by the synchronization means in a virtual space within the game.

[0155] In order to realize a game simulating a competition (soccer game), the game program of this embodiment causes a computer including an imaging device (camera) and a display device to execute an arrangement step of placing player characters in a virtual space in which a positional relationship with real space is defined and in which the virtual camera is arranged at a position in the virtual space corresponding to the position of the imaging device (camera) in the real space, a modification step of changing the angle of view of the virtual camera in accordance with the movement of the imaging device (camera) in real space, and a display step of superimposing a virtual image of the virtual space captured by the virtual camera on a real image of the real space captured by the imaging device and displaying it on the display device, and in the modification step, the angle of view of the virtual camera is narrowed the closer the virtual camera is to the area in which the player characters are placed.

[0156] For example, when the visual virtual agent determines that the user is running, the user's activity data indicates that the user is running, and the activity data can be obtained using a sensor such as a GPS tracker or a step sensor, and the visual virtual agent may be further configured to obtain user-specific data, for example, from the GPS tracker or the step sensor, and thus may be configured to monitor and automatically determine whether the user has stopped running. In this embodiment, under normal circumstances, each enemy is controlled to operate independently, and during a specific event (for example, a scene in which one of the enemies launches a specific attack), the enemy character control means causes the enemy character to launch an attack that changes the viewpoint position of the player character, and moves the enemy character that is within a specific range from the viewpoint position when changed by the attack into the field of view of the player character (=virtual camera) as viewed from the viewpoint position.

[0157] Furthermore, in the "flight mode", when the left foot position and the right foot position exceed the front boundary and the position of the tracking sensor attached to the head or the tracking sensor attached to the upper body other than the head moves to a rear position relative to the initial position when the game is started, that is, when the head or the upper body other than the head is tilted in the opposite direction to the direction in which the legs are extended, the moving speed and moving distance of the seating object and the virtual person may be increased to reflect the action. Here, in the camera control data, data from the start point to the end point is defined in a time series as shown in the figure, and if the data of the start point is defined by the time and the data of the intermediate points is defined by the time, by setting the position of the time as the playback position, it is possible to control the movement of the virtual camera along the route from the intermediate point of the movement route to the end point. When a virtual space including such an occluding object is captured by the virtual camera, the central part is displayed with an enlarged image of a part of the virtual space displayed in normal times, while the peripheral part is occluded by the occluding object, and the part of the virtual space displayed in normal times is not displayed (or is difficult for the user to see).

[0158] In this embodiment, a mechanism is described in which, when an HMD photographs a marker installed in a model room, a three-dimensional model corresponding to the marker is displayed. However, a mechanism may be used in which an action is detected and a three-dimensional model corresponding to the action is displayed. In addition, a mechanism in which a user who contributes more receives a larger reward is easy to understand from users, is fair in terms of the correlation between contribution and reward, and can prevent users from complaining within a group. Details of the mechanism of the MR technology are as described in the prior art described above. The mechanism of the MR technology is as described in the prior art documents and the prior art described above.

[0159] The wearer and / or caregiver may also be presented with the opportunity to personalize the user interface of the system as a reward system upon completion of several sessions of the various games offered. In this way, giving more opportunities to users who are good at the game leads to advantageous progress in cooperative matches in cooperative teams, and is likely to gain understanding and satisfaction from each user. Each seeking opportunity can be tracked by storing its location and the time it takes to look at it successfully. In one embodiment, the opportunity can be a match in which two or more users compete against each other.

[0160] Modifications In the above embodiment, an example was described in which the second position determination unit determines whether the first line or the second line intersects with a volume for hit determination having a relatively narrow third range set on the target object when the normal display control process is performed, and determines whether the first line or the second line intersects with a volume for hit determination having a relatively wide fourth range set on the target object when the special display control process is performed. However, the range for hit determination set on the first line and the second line may be changed without changing the range of the volume for hit determination set on the target object between the normal display control process and the special display control process. Also, the object generated in the distributor terminal may be generated in a virtual space in the distributor terminal, and information related to the generated object (for example, information normally required to display an object in a virtual space, such as the object's position, size, shape, polygon, etc.) may be sent to a server to generate the object in the virtual space in the server and move the position of the gift, or information related to the generated object may be sent to a viewer terminal to generate the object in the virtual space in the viewer terminal and move the position of the gift. In addition, in a state in which the target virtual object is determined to be pinched at multiple positions (typically, at one position it is determined to be pinched between multiple fingers of the left hand or FL5, and at the other position it is determined to be pinched between multiple fingers of the right hand or FR5), if the distance between any of the fingers determined to be virtually pinching the target virtual object at position and any of the fingers determined to be virtually pinching the target virtual object at position varies, the size of the target virtual object in the virtual space may be changed in accordance with the variation in distance.

[0161] Variations In the above-described embodiment, an example was given in which the special position calculation unit determines a special position based on the first designated position and the second designated position, and the display control unit controls an image based on the special position when a special display control process is performed. However, when the positional relationship between the first designated position and the second designated position is outside a given range, the display control unit may control the image based on the first designated position independently of the second designated position as a normal display control process, and may control the image based on the second designated position independently of the first designated position, and when the positional relationship between the first designated position and the second designated position is within a given range, the display control unit may control the image based on at least one of the first designated position and the second designated position as a special display control process. For example, when normal display control processing is performed, it may be determined whether a relatively narrow third range area with the first straight line as its center line or a relatively narrow third range area with the second straight line as its center line intersects with a hit determination volume having a predetermined range set on the target object, and when special display control processing is performed, it may be determined whether a relatively wide fourth range area with the first straight line as its center line or a relatively wide fourth range area with the second straight line as its center line intersects with a hit determination volume having a predetermined range set on the target object.

[0162] Thus, for example, hardware of an entertainment device operable to implement the mapping techniques described herein may be a computing device operating according to suitable software instructions, including a video data processor (such as a CPU and / or GPU) configured (e.g., by suitable software instructions) to obtain a first sequence of video images output by a video game title; a camera position data processor (such as a CPU and / or GPU) configured to obtain a corresponding sequence of in-game virtual camera positions from which the video images were created; a depth data processor (such as a CPU and / or GPU) configured to obtain a corresponding sequence of depth buffer values ​​for a depth buffer used by the video game during creation of the video images; and a mapping data processor (such as a CPU and / or GPU) configured (e.g., by suitable software instructions) to obtain, for each of a plurality of videos of the obtained sequence and corresponding depth buffer values, a mapping point corresponding to a selected predetermined set of depth values ​​corresponding to a set of predetermined positions within the respective video images, wherein for each pair of depth value and video image position, the mapping point has a distance from the virtual camera position based on the depth value, and a position based on the relative position of the virtual camera and each video image position, thereby obtaining a map dataset of mapping points corresponding to the first sequence of video images.A mapping method may thus include the steps of generating at least a portion of a first map including a graphical representation of some or all of the mapping points of at least a first map data set for display with a first video recording; generating at least a portion of a second map including a graphical representation of some or all of the mapping points of at least a second map data set associated with a different video recording of a different sequence of video images output by the same video game title and sharing the same in-game coordinate system; displaying at least a portion of the first map during playback of the first video recording, the displayed portion including at least a current virtual camera position associated with the displayed video images; and displaying at least a portion of the second map during playback of the first video recording if a respective portion of the second map is within a predetermined range of the current virtual camera position of the in-game coordinate system.

[0163] Thus, since the map points represent the endpoints of lines with links corresponding to depth information on the image, and directions corresponding to the relative position of the sample pixel on the screen and the optical axis of the virtual camera, as discussed above, whereas the map points represent lines emanating from the position of the virtual camera, each representing a distinct line of sight between the camera and the endpoints, it will be appreciated that color information for points along these lines is available, and color information for corresponding positions on any display surface below these lines can be sampled for each corresponding position in the map.

[0164] Therefore, the movement of the marker at a normal speed within the imaging range of the outer imaging unit can basically be tracked by updating the marker coordinate system and the relative relationship information.

[0165] Furthermore, for example, when a first finger determined to be facing ventrally and a second finger determined to be facing dorsally are both in contact with a target virtual object, and the distance between the coordinate value (x1, y1, z1) of the fingertip in the XYZ coordinate system of the virtual space of the fingertip and the coordinate value (x2, y2, z2) of the fingertip in the XYZ coordinate system of the virtual space of the fingertip is below a predetermined threshold value (at this time, the part of the target virtual object corresponding to the finger facing dorsally is deformed to be located behind the finger, but the position comparison in the Z-axis direction with the finger facing ventrally is performed using the coordinate value before deformation, so the finger is in a semi-transparent composite and drawn state), the information processing device may determine that the target virtual object is virtually sandwiched between the fingers at the position of the fingers (a predetermined range including (x1, y1) and (x2, y2), for example, the range of the smallest circle including these), and execute a predetermined process. A mapping method may thus include the steps of generating at least a portion of a first map including a graphical representation of some or all of the mapping points of at least a first map data set for display with a first video recording; generating at least a portion of a second map including a graphical representation of some or all of the mapping points of at least a second map data set associated with a different video recording of a different sequence of video images output by the same video game title and sharing the same in-game coordinate system; displaying at least a portion of the first map during playback of the first video recording, the displayed portion including at least a current virtual camera position associated with the displayed video images; and displaying at least a portion of the second map during playback of the first video recording if a respective portion of the second map is within a predetermined range of the current virtual camera position of the in-game coordinate system.

[0166] In the embodiment described above, when moving the first game space image displayed on the head mounted display, the transparency of the first game space image is set so that it gradually becomes more transparent from the center of the first game space image towards the edges, thereby making the boundary between the area where the first game space image is displayed and the area where the first game space image is not displayed less noticeable. However, for example, it is also possible to display a different image (for example, an abstract pattern or a pattern resembling countless balls and bats) in the area where the first game space image is not displayed, to give the impression that they are approaching.

[0167] Similarly, the hardware for the video playback apparatus may be a conventional computing device operating under suitable software instructions, such as a playback processor (e.g., a CPU operating under suitable software instructions) configured to acquire a first video recording of a video game play session including a sequence of video images, configured (e.g., by suitable software instructions) to acquire a second video recording generated by encoding a sequence of depth buffer values, and configured to acquire a sequence of in-game virtual camera positions associated with at least one of the acquired video recordings, and configured to provide the in-game virtual camera positions to an event analyzer for statistical analysis. the analysis processor (e.g., a CPU operating under suitable software instructions) configured (e.g., by suitable software instructions) to obtain data representing statistically significant in-game events and in-game event locations; a position calculation processor (e.g., a CPU operating under suitable software instructions) configured to calculate a location in a current video image of the first video recording that corresponds to the in-game event location responsive to an associated in-game virtual camera position and a decoded depth buffer value; and an augmentation processor (e.g., a CPU and / or GPU operating under suitable software instructions) configured to augment the current video image with a graphical representation of the statistically significant in-game event responsive to the calculated location.Finally, similarly, the event analyzer may be a server or a conventional computing device operating under suitable software instructions, including a receiving unit (e.g., an Ethernet port operating in cooperation with a CPU under suitable software instructions), an analysis processor (e.g., a CPU operating under suitable software instructions), a selection processor (e.g., a CPU operating under suitable software instructions), and a transmitting unit (e.g., an Ethernet port operating in cooperation with a CPU under suitable software instructions), wherein the receiving unit transmits the video game identifiers and one or more associated in-game events and their respective in-game locations from a plurality of video storage devices, The video game device is configured to receive a video game identifier and a selection processor configured to select one or more identified statistically significant in-game events that are associated with the received video game identifier and have an in-game location within a predetermined distance of the at least one received location, and the transmission unit is configured to transmit data representing the in-game events and their in-game locations to the video playback device.

[0168] In one example of this embodiment, it is not necessary for the character posture setting unit to determine the position coordinates of all the nodes of the virtual character in the virtual space using the positions of the user's body parts in the real space and the scaling coefficient, but the position of a preselected selected node in the virtual space among the nodes included in the bone model of the virtual character may be determined using the estimated positions of the user's body parts in the real space and the scaling coefficient, and then the positions of each node included in the bone model of the virtual character other than the selected node may be determined based on the position of the selected node determined in the virtual space. Also, an object generated in a distributor terminal may be generated in a virtual space in the distributor terminal, or information related to the generated object (for example, information normally required to display an object in a virtual space, such as the object's position, size, shape, polygon, etc.) may be sent to a server to generate the object in the virtual space in the server and move the position of the gift, or information related to the generated object may be sent to a viewer terminal to generate the object in the virtual space in the viewer terminal and move the position of the gift.

[0169] As a result, according to this aspect, when a user wearing a head mounted display needs to keep watching a predetermined object in a virtual space, such as an opponent in a fighting game or a racing car in a racing game, the user wearing the head mounted display can control the attitude of the head mounted display so that a virtual point is set at a position corresponding to the predetermined object in the virtual space, thereby making it possible to continuously display an area in the virtual space where the predetermined object exists on the display unit of the head mounted display. As described above, the object generated in the above-mentioned server device may be generated in the server device and the gift may be moved, or information related to the generated object (for example, information normally required to display an object in a virtual space, such as the object's position, size, shape, polygon, etc.) may be sent to a viewer terminal or a distributor terminal, and the object may be generated in the virtual space in the viewer terminal or distributor terminal, and the position of the gift may be moved. For example, all of the modules of the virtual camera control system may be included in a user terminal device (e.g., a desktop computer, a notebook computer, a workstation, a PDA, a web pad (particularly a smart pad), a mobile phone (particularly a smartphone), etc.) (not shown) that typically requires utilization and control of the virtual camera, but some of the modules may be included in other digital devices (e.g., a three-dimensional content providing server) (not shown) other than the user terminal device that can communicate with the user terminal device.

[0170] In this embodiment, the rendering camera setting unit obtains a predetermined (e.g., hard-coded in a program or read from a setting file) rendering camera position and information representing the direction of the field of view (e.g., vector information starting from the rendering camera position and passing through the center of the field of view) regardless of the position of the image sensor included in the camera, and sets these as the field of view information. Furthermore, the "device" may be in the form of specially designed hardware that is a general-purpose hardware specialized for executing software that enables the hardware to perform a specific function, software stored on a computer-readable medium, or a combination thereof. Output from the AI ​​system may be passed to a "hardwired" system (here, as part of a behavior processing engine), which can implement a set of rules and output a final social cue. The code module or any type of data may be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device, including a hard drive, solid-state memory, random access memory, read-only memory, optical disk, volatile or non-volatile storage device, combinations of the same, and / or the like.

[0171] Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that is capable of executing various functions by installing various programs.

[0172] It is also described that when a player points the controller at a virtual game space (pseudo-dimensional image) with an aiming cursor and then swings the controller up and down while pressing the A button, a second beam different from the first beam is shot toward the position in the virtual game space pointed to by the aiming cursor (or a weapon such as a chain with a chain attached to the end may be fired).When an online game realized by the game system is one in which a user operates one character selected from a plurality of characters and plays against other characters, the game progress information 42d may include at least one of character use information, character training progress information, and character setting information (appearance such as costume, weapon, hairstyle, facial expression parts, character name, skills, etc.). Here, the first state information includes at least one of the physical strength value or durability value of the first class object (when the first class object is a virtual person, the physical strength value of the first class object is displayed, and when the first class object is a virtual object such as a machine, a weapon, a vehicle, or the like, the durability value of the first class object is displayed), the number of virtual resources of the first class object, the defense value of the first class object, and the tool information of the first class object. The "action" here includes moving each part of the body, changing posture, changing facial expression, moving, speaking, touching or moving objects arranged in the virtual space, using weapons, tools, etc. held by the character, and the like. In addition, when a right-handed player plays a game with a weapon in his right hand, a screen configuration in which the character is displayed on the left side of the screen and other objects (map, enemies) are displayed on the right side of the screen as shown in the figure is considered to make gameplay easier for the right-handed player.

[0173] The system will preferably be configured to recognize that the user is holding an item and to understand the type of interaction the user is having with the item. Furthermore, in the case of a setting that does not allow the item to be dropped (an example of a performance in which the user jumps high and jumps onto something), the information processing device determines that the user's body is rising in real space if the difference between the real reference position information and the real reference position information Pp at the time of the previous execution, which is repeatedly executed at a predetermined timing, is P-Pp>0, and executes the same processing as described above. In addition, since it may be difficult to operate if the inertial coordinate system moves during operation, the on / off of this control mode may be controlled so that the user performs an input operation using, for example, an input controller, or may be controlled to be off when an operation is performed on a virtual object. Alternatively, when the generation unit is transmitted motion information indicating that the user has made a motion such as kicking something, it generates image information (image) showing the user kicking an object present on the trajectory of the user's foot in the virtual reality space or the augmented reality space. In this embodiment, the system is described as one in which the HMD photographs a marker installed in a model room and displays a 3D model corresponding to the marker, but the system may also detect some action and display a 3D model corresponding to that action.

[0174] The method of the server determining whether or not a second object exists between the target virtual object and the target virtual resource includes, but is not limited to, acquiring a connecting line between the target virtual object and the target virtual resource based on the first reference coordinates and the second reference coordinates, determining whether or not a second object exists on the connecting line, and determining that a second object does not exist between the target virtual object and the target virtual resource if the second object does not exist on the connecting line. Here, the target pace extracted from the target pace database may be generated based on the running distance and running time of the latest running among the training data of running performed by the user in the past, or may be generated based on the running distance and running time extracted based on various conditions such as the topographical information and weather information input by the user when the topographical information (elevation, course shape, etc.) of the running course and the weather information (temperature, wind direction, wind speed, etc.) are stored in association with each other as the training data in the target pace database.

[0175] Then, in step 2, when the running speed reaches the target value, the CPU causes the image generating unit to enlarge the display size to a preset standard size so that the virtual runner appears to be running just in front of the user's field of vision, and generates a moving image (standard image) of the virtual runner synchronized with the user's motion state by matching the timing of the virtual runner's feet to the timing of the user's feet and matching the pitch of the virtual runner to the user's pitch based on the user's pitch and foot take-off and landing timing. Here, the display command is for generating first visual information from data of the first class object included in the display command, generating second visual information from data of the second class object included in the display command, and instructing the terminal to display the first visual information on the user interface or to display the first visual information and the second visual information when the first class object does not exist within the field of vision range of the visual angle corresponding to the target virtual object, and the amount of information of the first visual information is greater than the amount of information of the second visual information.

[0176] Then, in a step, when the running speed reaches the target value, the CPU causes the image generation unit to generate a moving image of the virtual runner with an enlarged display size so that the virtual runner appears to be running directly in front of the user's field of vision, and based on the user's pitch and foot take-off and landing timing, matches the take-off and landing timing of the virtual runner to the take-off and landing timing of the user and matches the pitch of the virtual runner to the user's current pitch, thereby generating a moving image of the virtual runner synchronized with the user's motion state.

[0177] For example, in the case of a virtual space simulating a baseball field, one of the "first virtual space image" and the "second virtual space image" may be an image showing the virtual space as seen from the position of one of the player characters (e.g., a catcher character or a batter character) or an umpire character (e.g., a head umpire character), and the other may be an image showing the virtual space as seen from another position (e.g., the position of the stands). In the case of a full-scale baseball game rather than a baseball board game, if the first user wants to curve a virtual object thrown by a first player character to the right, the left controller is used to input a direction to the right (e.g., an input to tilt the analog stick to the right), and as a result, the virtual object moves on a trajectory (curve ball, slider ball) that curves to the left as seen by the right-handed first player character. Generally speaking, the method for executing the game program of the baseball game by the server includes the steps of grouping a plurality of users who cooperate in a cooperative match based on at least one of the play history in a competitive game of each user or the players owned by each user, generating a cooperative team including at least one player owned by each of the grouped users, and progressing a match between the cooperative team and the opposing team. Specifically, for example, in the case of a virtual space simulating a baseball field, an image showing the virtual space in a state where a first team is playing defense may be a "first virtual space image," and an image showing the virtual space in a state where a second team is playing defense may be a "second virtual space image."

[0178] In other words, if the input device is lowered, such as in a golf swing stance, the candidate surface calculated using the Y' axis may be more relied upon to calculate a new reference plane, and if the input device is raised, such as in a baseball batter's stance, the candidate surface calculated using the Z axis may be more relied upon to calculate a new reference plane.

[0179] Variations In the above-described embodiment, an example was given in which the second position determination unit determines whether or not the first line or the second line intersects with a hit determination volume having a relatively narrow third range set on the target object when a normal display control process is performed, and determines whether or not the first line or the second line intersects with a hit determination volume having a relatively wide fourth range set on the target object when a special display control process is performed. However, it is also possible to change the hit determination ranges set for the first and second lines without changing the range of the hit determination volume set on the target object between the normal display control process and the special display control process.

[0180] Variations In the above-described embodiment, an example was given in which the special position calculation unit determines a special position based on the first designated position and the second designated position, and the display control unit controls an image based on the special position when a special display control process is performed. However, when the positional relationship between the first designated position and the second designated position is outside a given range, the display control unit may control the image based on the first designated position independently of the second designated position as a normal display control process, and may control the image based on the second designated position independently of the first designated position, and when the positional relationship between the first designated position and the second designated position is within a given range, the display control unit may control the image based on at least one of the first designated position and the second designated position as a special display control process. For example, when normal display control processing is performed, it may be determined whether a relatively narrow third range area with the first straight line as its center line or a relatively narrow third range area with the second straight line as its center line intersects with a hit determination volume having a predetermined range set on the target object, and when special display control processing is performed, it may be determined whether a relatively wide fourth range area with the first straight line as its center line or a relatively wide fourth range area with the second straight line as its center line intersects with a hit determination volume having a predetermined range set on the target object.

[0181] Variations In the above-described embodiment, an example was given in which the second position determination unit determines the positional relationship between the target object and the first designated position when the first mark and the second mark are displayed, and performs a normal position determination process to determine the positional relationship between the target object and the second designated position, and performs a special position determination process to determine the positional relationship between the target object and the special position when a special mark is displayed. However, the second position determination unit may determine the positional relationship between the target object and the first designated position, and determine the positional relationship between the target object and the second designated position, even when a special mark is displayed.

[0182] Variations In the above-described embodiment, an example has been described in which the update unit updates a given parameter in a first update mode when a normal display control process is performed, and updates the given parameter in a second update mode when a special display control process is performed. However, the display device may further include an attribute setting unit that sets one of a plurality of attributes in association with each of the first and second designated positions, and an update mode change unit that changes the update mode of at least one of the first and second update modes based on a combination of attributes associated with each of the first and second designated positions.

[0183] In addition, it has been very difficult to secure a person who meets the abilities and conditions required for a pacemaker, that is, a runner with the same or higher running ability as the user and who can understand the user's ability and provide appropriate guidance. However, by applying the above-mentioned exercise support device and exercise support method, a pacemaker that is optimal for oneself can be easily used. The biorecovery ability data can include at least one of the following: data that is generally applicable according to the gender, age, and athletic event in progress of the athlete, and personal data that is data measured during the existing competition history or training performed by each athlete and accumulated individually. In the generating step, the order is determined in consideration of attributes linked to each athlete, and the attributes may include at least one of a parameter representing the athlete's ability and a role in the team set for the athlete. This is because, in order to contribute a desired athlete with high ability to the cooperation team, it is more effective to evenly place players with a certain ability or higher in all positions in the deck than to biasedly strengthen a specific player in a specific position. For this reason, when a character belonging to the user team is in possession of the ball, the evaluation value calculation unit may increase the evaluation value of a partial area in which an ally character whose offensive ability is higher than the standard (i.e., whose offensive ability parameter value is higher than the standard value) is located.

[0184] [Processing for Referencing the Type of Virtual Object Set for Other Player Attribute Information] Note that in the above example, when there are multiple objects in real space that are expected to be replaced with a predetermined type of virtual object (NPC, etc.), it is decided at random for each group identifier (player attribute information) which object to replace with the predetermined type of virtual object and place. However, this embodiment is not limited to this, and the server device may obtain information such as whether or not there is a group identifier that is currently being replaced with the predetermined type of virtual object and placed, or how many such group identifiers there are for each object, and may refer to the information to select objects that are not currently being replaced with the predetermined type of virtual object for other group identifiers, and may replace the selected object with the predetermined type of virtual object and place it.

[0185] [User] In the above examples, the amount of movement in the virtual space is determined based on the amount of movement of an object (which may be a part of the user) that moves with the user, but this embodiment is not limited to this. The information processing device may use the amount of movement of an object that does not move with the user, such as a drone that moves in the real space, and may determine the amount of movement of a virtual object in the virtual space corresponding to the object, such as a drone, by multiplying the amount of movement in a predetermined axis direction by a predetermined factor, and may move the virtual object by the determined amount of movement and present an image representing the state in which it has been moved to the user. This virtual environment may be used to simulate an actual environment in reality, and for example, this virtual environment may include the sky, land, or sea, and the land may include environmental elements such as a desert or a city, and the user may control the virtual object to move in this virtual environment, and this virtual object may be a virtual avatar that represents the user in this virtual environment, and this virtual avatar may be any form, such as a human or an animal, and this application is not limited to this. In another example of this embodiment, instead of this calculation, when determining position coordinates in the virtual space corresponding to the user's focus area which is located at the real reference position in the real space, the sign of the relative position from the real reference position to the foot position (information determined by the processing in the figure) is inverted, multiplied by a predetermined ratio, and further converted to the scale of the virtual space to obtain information on the relative position to the real reference position as seen from the foot position (a value in the world coordinate system of the virtual space), and the information on the virtual reference position determined by the processing in the figure is added to this to determine position coordinates in the virtual space which correspond to the user's focus area which is located at the real reference position in the real space.

[0186] Furthermore, in the above embodiment, the angular difference (|60 degrees - 30 degrees| = 30 degrees) between the controller pitch angle at which the virtual camera's pitch rate is 0 (the shooting direction is horizontal) and the controller pitch angle corresponding to the upper limit of the virtual camera's range of motion was the same as the angular difference (|0 degrees - 30 degrees| = 30 degrees) between the controller pitch angle at which the virtual camera's pitch rate is 0 and the controller pitch angle corresponding to the lower limit of the virtual camera's range of motion; however, this embodiment is not limited to this, and the angular difference between the controller pitch angle at which the virtual camera's pitch rate is 0 and the controller pitch angle corresponding to the upper limit of the virtual camera's range of motion may be greater or smaller than the angular difference between the controller pitch angle at which the virtual camera's pitch rate is 0 and the controller pitch angle corresponding to the lower limit of the virtual camera's range of motion.

[0187] The server may then extract skills that the user lacks with respect to the target state based on the skill proficiency status and characteristics obtained by the analysis, and may present the trainee with advice (e.g., advice on correcting form such as "keep your wrists higher," "don't raise your shoulders," "don't move your head back and forth," and "don't round your back") and practice menus (e.g., practice menus and practice schedules for improving the lacking skills) based on the extracted results. For example, if a heat reduction skill (second attribute information) that can halve the effect of the attribute "high temperature" is associated with the player character, and an equipment object attached to the player character or hand object is associated with an equipment effect (third attribute information) that can halve the effect of the attribute "high temperature," the processor may add up the skill and the equipment effect and determine that the effect of the attribute "high temperature" can be nullified. In the case where the online game realized by the game system is one in which the user operates one character selected from a plurality of characters and plays against other characters, the game progress information may include at least one of character use information, character training progress information, and character setting information (appearance such as costume, weapon, hairstyle, facial parts, character name, skill, etc.). Such skill exploration practice can be said to be a process of trial and error exploration of how to use the body, posture, how to direct attention, and how to use the senses (what to listen to, when and where to look, etc.) to reduce unnecessary muscle tension during playing, reduce mistakes, and create the desired sound and expression. For example, if skill B and skill B (for example, skill B and skill B are skills (skills) for playing different melodies) are practiced consecutively with the same amount of practice, the establishment of skill B may hinder the establishment of skill B, and as a result, only skill B may be established.

[0188] Thus, advantageously, mapping points or map data sets can be included for at least a portion of the map corresponding to image data not yet displayed in such a way that a playback device can access the mapping points, and optionally altitude data, color data, etc. (i.e., access map information corresponding to a current position shown in the video, but also image / depth / camera information for positions before or after the current position that would otherwise be required). In other implementations, the wearable system can determine context coefficients associated with objects in the user's environment and enable automatic cone updates, in addition to or as an alternative to manual updates, which can advantageously make it easier for the user to interact with objects in the environment, since less user input is required. Note that the reward points are not limited to the above examples, and may be experience points for increasing the user's level, virtual currency that can be exchanged for various items in the game, or other in-game value used to gain an advantage in the game. In this way, when the parameters used in the game calculations are set to values ​​that are more advantageous to the player character in the game than in the normal state, the player character may not be hit by an attack that would normally hit him, or may not receive damage even if it is hit, or may receive less damage than in the normal state.The special points, ... are locations in the virtual space that are advantageous in the game, and may be set in positions that are in the shadows of defensive wall objects such as walls, buildings, trees, cars, and rocks, making them less susceptible to enemy attacks.

[0189] Here, the camera control data is defined in a time series from the start point to the end point as shown in the figure, and if the start point data is defined by the time and the intermediate point data is defined by the time, the virtual camera can be controlled to move from the intermediate point of the movement route to the end point by setting the time position as the playback position. Also, as an example, if at least one of the object position, object orientation, virtual camera position, and virtual camera orientation changes during the operation of transforming the object, parameters for scaling the object may be calculated by changing any of the object's scaling axis, expansion direction, center position, etc. according to the state of the object and / or virtual camera after the change. As another example, even if at least one of the object position, object orientation, virtual camera position, and virtual camera orientation changes during the operation of transforming the object, the process of scaling the object may be continued using the state of the object and / or virtual camera before the change. Also, for example, in the process of sequentially referring to the luminance values ​​of pixels from pixel P(n-8) toward the left to find a local maximum value of luminance values ​​(the first local maximum value), if the luminance value of the reference pixel exceeds a predetermined value (e.g., 250), the process of finding the local maximum value of luminance values ​​may be interrupted and a white area luminance value may be calculated based on the luminance value of the reference pixel. Also, if the grab position goes outside the display range of the display during an operation to transform an object, the process of changing the enlargement or reduction amount may be stopped at the point when the grab position goes outside the display range (i.e., the size immediately before the grab position goes outside the display range is maintained).

[0190] From the viewpoint of efficiently burning fat through aerobic exercise, it is most important to take oxygen into the blood, and care must be taken not to over-pace and become anaerobic. The HP system may use data from both the FDT and the VIO to increase the accuracy, speed, and / or efficiency of the HP system in determining head posture, angular velocity, and / or linear velocity. For example, a mathematically definable surface may be an efficient (e.g., compared to a large-scale point cloud) and digestible input to other processing devices such as a game engine. For example, a mathematically definable surface may be an efficient (e.g., compared to a large-scale point cloud) and digestible input to other processing devices such as a game engine. However, in order to perform efficient training, it is important to determine the scenes to focus on in accordance with the player's movements.

[0191] The synchronization frequency determination unit sets the synchronization frequency to a first frequency when the second user is present within a predetermined angular range including the line of sight of the first user in the virtual reality space and closer than a predetermined distance from the first user, sets the synchronization frequency to a second frequency lower than the first frequency when the second user is present within the predetermined angular range but farther than a predetermined distance in the virtual reality space, and sets the synchronization frequency to a third frequency lower than the first frequency when the second user is present outside the predetermined angular range in the virtual reality space.

[0192] The synchronization frequency determination unit sets the synchronization frequency high when the second user is within a predetermined angle range including the line of sight of the first user in the virtual reality space and closer than a predetermined distance from the first user, and sets the synchronization frequency low when the second user is within a predetermined angle range and farther than a predetermined distance in the virtual reality space. That is, the synchronization frequency determination unit sets a first delivery frequency for users and U3 who are in the near area within the field of view, sets a second delivery frequency for users and U5 who are in the medium area within the field of view, sets a third delivery frequency for users to U9 who are in the far area within the field of view, and sets a fourth delivery frequency for users Uj (j≧10) who are in the out-of-field area. Based on the user's answer result, if the user has relatively little exercise experience (the period of time that the user has played sports is less than a certain period, the habit of exercising is less than a predetermined frequency, etc.), the server may suggest to the user training items included in the training item information that have a relatively small load on the body or a relatively low level of difficulty. In the above example, once a background voxel is placed, it is not deleted thereafter. However, it is also possible to check for the presence of a subject portion corresponding to the background voxel at a lower update frequency than the update process for person voxels, and delete background voxels that are placed at candidate positions where the presence of a subject portion is no longer detected.

[0193] The embodiment of the present application mainly relates to electronic games or analog training scenes, and takes an electronic game scene as an example. A user can perform actions in advance on the electronic device, and the electronic device can download a game setting file for the electronic game after detecting the user's actions. The game setting file may include an application program, interface display data, or virtual environment data for the electronic game, and the user can call up the game setting file and render and display the electronic game interface when logging in to the electronic game on the electronic device. In this embodiment, the main unit performs a correction to reduce tension in the vertical direction, but in other embodiments, a correction to reduce tension in any direction according to the game content or game settings may be performed. The position coordinates of the viewpoint and the position coordinates of the virtual camera do not necessarily need to be completely the same, and there may be a slight difference (for example, a difference of about 10%) between the position coordinates of the two within the range of adjustment of the game settings. The coefficient and the coefficient may be set appropriately according to the game content or game settings, and the coefficient may be set larger or smaller than the coefficient. For example, the game setting means performs a process for requesting the server device to register a user, and a process for requesting the server device to delete information that can identify the user.

[0194] Here, the game processing unit performs various game processes such as coin (cost) acceptance process, various mode setting process, game progress process, selection screen setting process, process for determining the position and rotation angle (rotation angle around X, Y or Z axis) of an object (one or more primitive surfaces), process for moving an object (motion process), process for determining the position of the viewpoint (position of the virtual camera) and the line of sight angle (rotation angle of the virtual camera), process for arranging objects such as map objects in the object space, hit check process, process for calculating the game result (achievement, achievement), process for multiple players to play in a common game space, and game over process, based on operation data from the operation unit, personal data from the save information storage device, game program, etc. In addition, in this case, by registering a panoramic image, the user may be given a privilege (for example, coins that can be used in the game, special items, a predetermined parameter increase, etc.). Furthermore, the amount of coins given to the user is specified based on, for example, the degree of support of the user for the player.

[0195] More specifically, the image recognition process may be a process of calculating the position of the face, eyes, and mouth of the player as a result of the above process, judging the player's facial expression (whether the player is smiling, talking, or blinking), or a process of performing personal authentication (identifying the player as a previously registered person). When the user first decides the angle (position of the virtual camera) at which they want to shoot while viewing the content (panoramic video) in the virtual space, data on the timing of shooting the panoramic video, such as who wants to shoot from what position and when (or when the video was shot), is obtained. The angle information (for example, information indicating who shot from what position, at what time, and who shot) is associated with the ID of the content and accumulated in the server for each content. Alternatively, a configuration may be adopted in which a real user issues an invitation in the real space, and information on who was invited by whom is added to the user information.

[0196] The present embodiment further relates to a computer-implemented method for generating a visual virtual agent configured to perform an artificial physiological color change response to a user for at least one visual virtual agent state and / or for at least one state of a user, the visual virtual agent being configured to perform a visual response to a user, the at least one visual virtual agent state and / or the state of the user being determined based on user specific parameters including user activity data and / or user physiological and psychological data, the at least one user specific parameter being acquired using at least one sensor, the visual response being dependent on changes in the user activity data and / or the user physiological and psychological data, the visual response at least partially comprising a color change of the visual virtual agent for the at least one determined visual virtual agent state and / or the at least one determined state of the user, the method comprising: acquiring, using at least one sensor, the user activity data and / or the user physiological and psychological data over time; acquiring, from the at least one sensor, the user activity data and / or the user physiological and psychological data over time; acquiring user activity data and / or user physiology with a processor, analyzing the user activity data and / or the user physiology with a processor, determining a visual virtual agent state based on the user acquired activity data and / or the user acquired physiology with a processor and / or determining a user's current state based on the received acquired activity data and / or the user acquired physiology with a processor, selecting at least one sensor for generating a color change response based on the user determined current state and / or based on the determined visual virtual agent state, depicting with a graphics processing unit the visual virtual agent in a first color for the determined visual virtual agent state and / or the user determined current state, displaying on a display device, on at least one display, the visual virtual agent in a first color for the determined visual virtual agent state and / or the user determined current state, using the selected at least one sensor,The method includes the steps of: acquiring (in real time) by a processor user activity data and / or user physiological and psychological data for the determined visual virtual agent state and / or the user's current state; analyzing (in real time) by a processor the user acquired activity data and / or the user's physiological data from the selected at least one sensor; determining by a processor a change in the user acquired activity data and / or the user's acquired physiological data; rendering by a graphics processing unit the visual virtual agent in a second color in response to the determined change in the user acquired activity data and / or the user's acquired physiological data; and displaying on a display device the visual virtual agent in the second color for the determined visual virtual agent state and / or the determined current state of the user;

[0197] Additionally or alternatively, the wearable device may use distance, size of periocular features, and / or other factors to determine whether the wearable device is in place or removed. The body may have a sponge material that can easily evacuate sweat and heat from the athlete and absorb external shocks, and a portion of the body may be waterproofed to protect the terminal from moisture caused by sweat from the athlete and environmental factors such as rain. These and / or other factors may be used alone or in combination to determine whether movement has occurred, is occurring, or will occur. These exemplary factors may be used alone or in combination to determine the relative position between the HMD and the user's face. These factors may create additional difficulties for individuals as they may not mirror and establishing a connection with other people may be more difficult. There are various methods of aerobic exercise, such as jogging and swimming, but walking is particularly popular because it is easy to do.

[0198] If it is possible in terms of the processing capacity of the display control device, a method for updating the first virtual space image displayed on the head mounted display in accordance with the orientation of the user's head may be to newly acquire a first virtual space image in a viewing direction corresponding to the changed orientation of the head mounted display, as with the second virtual space image, and update the first virtual space image displayed on the head mounted display to the newly acquired first virtual space image, but this method requires the first virtual space image to be newly acquired along with the second virtual space image, which increases the processing load. A method for updating the first game space image displayed on the head mounted display in accordance with the orientation of the user's head may be to newly acquire a new first game space image representing the game space in a viewing direction corresponding to the changed orientation of the user's head, as with the new second game space image, and update the first game space image displayed on the head mounted display to the newly acquired new first game space image, but this method requires the new first game space image to be newly acquired along with the new second game space image, which increases the processing load. As a method for updating the first game space image displayed on the head mounted display to match the orientation of the user's head, a method can be considered in which, similar to the new second game space image, a new first game space image corresponding to the changed orientation of the head mounted display is newly acquired, and the first game space image displayed on the head mounted display is updated to the newly acquired new first game space image; however, this method increases the processing load because it is necessary to newly acquire a new first game space image along with the new second game space image.Pe(t=n): Position of the second user to be corrected this time (current corrected gaze information) Pe(t=n-1): Position of the second user corrected last time (previous corrected gaze information) Ps(t=n-1): Position of the second user received last time (previous gaze information) Re(t=n): Head direction of the second user to be corrected this time (current corrected gaze information) Re(t=n-1): Head direction of the second user corrected last time (previous corrected gaze information) Rs(t=n-1): Head direction of the second user received last time (previous gaze information) Fs: Weighting function: Weighting function: Weighting function: Weighting functionHere, the position of the second user and Ps are expressed in three-dimensional coordinates in virtual reality space. The server may then extract the skills that the user lacks with respect to the target state based on the skill proficiency and characteristics obtained through the analysis, and based on the extracted results, present the trainee with advice (e.g., advice on correcting form such as "keep your wrists higher," "don't raise your shoulders," "don't move your head back and forth," and "don't round your spine") or a practice menu (e.g., a practice menu or practice schedule for improving the lacking skills).

[0199] In addition, when resetting the orientation of the virtual camera, only the yaw angle is reset because if the pitch angle and roll angle are also reset, the orientation of the virtual camera in the virtual space cannot be changed to correspond to the change in the orientation of the main device in the real space. When it is not determined in step that the remaining time of the cool time set for the tapped portal is 0 minutes, or when it is not determined in step that the distant play flag associated with the portal is reset, the process proceeds to step and the ticket consumption inquiry screen shown in the figure is displayed superimposed on the touch screen. For example, when the selection of the target object is completed in the game progress process (in the case of step), when it is determined that the mission clear condition is established (in the case of step), etc., the elapsed time measured to provide information may be reset. In this embodiment, the orientation of the virtual camera is reset, but the background object in the virtual space may be moved so that the horizontal component of the vector indicating the direction of the line of sight coincides with the reference direction. To reduce deviations, the gyroscope is periodically reset, and the resetting may be performed using other available information such as position / orientation determination based on visual tracking of the object, an accelerometer, a magnetometer, etc.

[0200] That is, for example, when tree polygon models arranged as fixed objects in a field are calculated from model position information of a tree arrangement file to be model numbers 1, 2, 3, 4, 5 ..., based on the arrangement position data of the virtual camera, when the virtual camera is at position, the number of tree polygon models to be selectively acquired as images on the projection plane will be three, the model number and 3, when the virtual camera is at position, the number of tree polygon models to be acquired as images on the projection plane will be three, the model number and 4, when the virtual camera is at position, the number of tree polygon models to be acquired as images on the projection plane will be three, the model number and 5. For example, when the corresponding relative relationship information is not acquired in the current frame, but the position where the marker should be is not included in the captured image, there is a possibility that the marker is not removed simply because it is not captured in the captured image, and therefore the deletion unit does not delete the marker coordinate system. When placing a deselected component on a background object, the image generating unit may determine whether or not the placement position specified by the controller object (the position where the component is originally supposed to be placed) coincides with a prohibited placement position, in other words, whether or not the position is included in an area indicated by the prohibited placement position. In the above-described processes or operations, the processes or operations can be freely changed as long as no inconsistencies in the processes or operations occur, such as the use of data that should not yet be available in a certain step. For example, an object showing a center circle that should have been along the circumference of a marker is not along the circumference.

[0201] Various aspects A computer program relating to the first aspect can "when executed by at least one processor, cause at least one processor to function as follows: acquire motion information regarding the motion of each of multiple parts of a broadcaster's body present in physical space output from multiple first sensors attached to the multiple parts of the body; acquire position information regarding the position of the body output from at least one second sensor attached to the broadcaster's body; calculate the position of each of the multiple parts in physical space based on the motion information; and calculate the position of the body in physical space based on the position information."

[0202] The online game image may further include various other objects such as game characters such as NPCs (non-player characters) appearing in the online game, scenery such as buildings and trees, furniture in a room and interior decoration of walls, items lying on the ground, parts of the body (hands and feet) of the avatar (the user himself) operated by the user, and objects (guns and swords) held by the avatar. Various information managed by the server includes the position, posture, and shape of real objects in a virtual space that is aligned with the real space and superimposed thereon (or a space where the virtual space is aligned with the real space and superimposed thereon = MR space (in other words, a space where the real space and the virtual space are combined)). Note that the points as rewards are not limited to the above examples, and may be experience points for raising the user's level, virtual currency that can be exchanged for various items in the game, or other in-game values ​​used to advance the game advantageously. Alternatively, an information processing device may read markers or QR codes (registered trademark) stored in facilities at various locations, transmit the read data to a search server, and the search server may perform a database search based on the received data and provide access information for obtaining information in the vicinity of the location where the received marker or QR code (registered trademark) is stored.

[0203] In the above example, the necessity of controlling the upward movement of the virtual camera in response to the entry into the low-angle area may be determined according to the type of character (for example, the type of clothing worn, etc.). Here, the characters are assumed to be, for example, a monster that is cursed and imprisoned, and an avatar that speaks and moves in response to the voice and action of the actor. For example, the input device may be something that the player holds and moves, or it may be something that the player wears and moves. The game medium may be, for example, an item such as clothing that can be worn by the character. The object may be a simple character icon that represents a talent or a fan with a two-head body, or may be an elaborate character icon. [Explanation of symbols]

[0204] 10 Virtual space movement device 11. Shoes 12 Odor generating device 13 Characters 14 Travel route information 15. Accessory Information 100 Storage section 200 Acquisition Department 300 Calculation Unit 400 Control section 500 Output section 600 units purchased 700 direction indicator 800 Connection section

Claims

1. A storage unit that stores movement course information including scenery when a character moves in a virtual space in cooperation with a user; An acquisition unit that acquires biological information indicating a biological state of a user who is exercising and exercise information indicating an amount of exercise of the user; a calculation unit that calculates a virtual transformation distance in the virtual space based on the motion information; a control unit that changes a viewpoint in the virtual space by moving a virtual camera in the virtual space from a first position to a second position according to the virtual transformation distance; an output unit that outputs the movement course information and the character; the control unit changes the viewpoint and changes a light source setting of movement course information output to the output unit, and includes a direction instructing unit to instruct the user, via the output unit, on a movement course information along which the character has moved, as a direction of movement of the user; The control unit of the virtual space movement device designates a character based on the bio-information and motion information, and displays a character with a warning meaning when the bio-information becomes a physically abnormal value.

2. 2. The virtual space movement device according to claim 1, wherein the control unit changes the viewpoint and controls sound effects output to the output unit.

3. the virtual space movement device includes shoes that apply impacts to soles of the user's feet; 3. The virtual space movement device according to claim 2, wherein the control unit controls the type, magnitude, and timing of the impact of the shoes based on the movement course information.

4. The virtual space movement device according to claim 1 , wherein the motion information includes any one of a walking pace, a number of steps, and a posture acquired when the user actually walks.

5. The computer a storage step of storing movement course information including scenery when the character moves in the virtual space in cooperation with the user; An acquisition step of acquiring biological information indicating a biological state of a user who is exercising and exercise information indicating an amount of exercise of the user; a calculation step of calculating a virtual transformation distance in the virtual space based on the motion information; a control step of changing a viewpoint in the virtual space by moving a virtual camera in the virtual space from a first position to a second position according to the virtual transformation distance; an output step of outputting the movement course information and the character; the control step includes a direction instructing step of changing the viewpoint and changing a light source setting of movement course information outputted in the output step, and instructing the user on a movement direction of the user via the output step based on the movement course information along which the character has moved; The control step specifies a character based on the vital information and the motion information, and, if the vital information becomes a physically abnormal value, displays a character with a warning meaning.

6. On the computer, A memory function that stores information about a course of movement of a character, including scenery, when the character moves through a virtual space in cooperation with a user; An acquisition function for acquiring biological information indicating a biological state of a user who is exercising and exercise information indicating an amount of exercise of the user; A calculation function for calculating a virtual transformation distance in the virtual space based on the motion information; a control function of changing a viewpoint in the virtual space by moving a virtual camera in the virtual space from a first position to a second position according to the virtual transformation distance; an output function for outputting the movement course information and the character; the control function includes a direction instruction function for changing the viewpoint and changing a light source setting of movement course information output to the output function, and instructing the user on a movement direction of the user via the output function based on the movement course information along which the character has moved; The control function is a virtual space movement program that designates a character based on the bio-information and movement information, and displays a character with a warning meaning when the bio-information becomes a physically abnormal value.

Images