Information processing device, method, and program

Abstract

An information processing device according to one embodiment of the present invention comprises: a detection unit that, by using a captured image of a user riding a bicycle, detects the positions of both of the user's shoulders and the position of one of the user's feet; a speed calculation unit that calculates a travel speed of a bicycle ridden by the user in a virtual space on the basis of the position of the user's one foot at a certain time as detected by the detection unit; a direction calculation unit that calculates the slope of a line segment connecting the positions of the user's shoulders detected by the detection unit, and calculates a direction of advance of the bicycle ridden by the user in the virtual space on the basis of the calculated slope; and a generation unit that generates video in which the bicycle ridden by the user in the virtual space travels at the travel speed calculated by the speed calculation unit and in the direction of advance calculated by the direction calculation unit.

Description

Information processing device, method and program

[0001] FIELD Embodiments of the present invention relate to an information processing device, method, and program.

[0002] Virtual cycling, where you ride a bicycle in a virtual space, mainly involves setting up specialized equipment to control an avatar in the virtual space. Examples of commercially available services that use this technology include Zwift (registered trademark). For example, the user's movements are transmitted to a server via various sensors, including a handlebar tracking controller, bicycle tilt sensor, wheel rotation sensor, pedal rotation sensor, and resistance unit. The server then calculates and displays a video of the user's movements on a display.

[0003] Development of Virtjal Reality Cycling Simulator / Schramka, Filip; Arisona, Stefan; Joos, Michael; Erath, Alexander Publication

[0004] However, the above-mentioned method requires the effort of preparing multiple dedicated devices, which is generally not available to most users, and is expensive, making it difficult for general users to use.

[0005] This invention was made in light of the above-mentioned circumstances, and its purpose is to provide an information processing device, method, and program that can output images of bicycles traveling in a virtual space with high versatility.

[0006] An information processing device according to one aspect of the present invention comprises a detection unit that detects the positions of both shoulders and one foot of a user using an image captured while the user is riding a bicycle; a speed calculation unit that calculates the traveling speed of the bicycle being ridden by the user in a virtual space based on the position of one foot of the user at a certain time detected by the detection unit; a direction calculation unit that calculates the slope of a line segment connecting the positions of both shoulders of the user detected by the detection unit and calculates the traveling direction of the bicycle being ridden by the user in the virtual space based on the calculated slope; and a generation unit that generates an image of the bicycle being ridden by the user in the virtual space traveling at the traveling speed calculated by the speed calculation unit and in the traveling direction calculated by the direction calculation unit.

[0007] An information processing method according to one aspect of the present invention is a method performed by an information processing device, and includes: a detection unit of the information processing device detecting the positions of both shoulders and one of the user's feet using an image captured when the user is riding a bicycle; a speed calculation unit of the information processing device calculating the traveling speed of the bicycle ridden by the user in a virtual space based on the position of one foot of the user at a certain time detected by the detection unit; a direction calculation unit of the information processing device calculating the slope of a line segment connecting the positions of both shoulders of the user detected by the detection unit and calculating the traveling direction of the bicycle ridden by the user in the virtual space based on the calculated slope; and a generation unit of the information processing device generating an image of the bicycle ridden by the user in the virtual space traveling at the traveling speed calculated by the speed calculation unit and in the traveling direction calculated by the direction calculation unit.

[0008] According to the present invention, it is possible to output a highly versatile image of a bicycle traveling in a virtual space.

[0009] FIG. 1 is a diagram illustrating an example application of a virtual cycling system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of the positions of both shoulders and one foot of a user. FIG. 3 is a block diagram illustrating an example functional configuration of an information processing device of the virtual cycling system according to an embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a recording result of the position of one foot of a user. FIG. 5 is a diagram illustrating an example of a result of sorting the recording result of the position of one foot of a user. FIG. 6 is a diagram illustrating a first example of a recording result of the position of one foot of a user used to calculate the speed of a bicycle. FIG. 7 is a diagram illustrating a second example of a recording result of the position of one foot of a user used to calculate the speed of a bicycle. FIG. 8 is a diagram illustrating an example of a calculation of the tilt of both shoulders of a user. FIG. 9 is a diagram illustrating an example of changes in the viewpoint and field of view in a virtual reality (VR) space. FIG. 10 is a diagram illustrating a first example of a calculation of tilt. FIG. 11 is a diagram illustrating a second example of a calculation of tilt. FIG. 12 is a diagram illustrating a third example of a calculation of tilt. FIG. 13 is a diagram illustrating a fourth example of a calculation of tilt. FIG. 14 is a diagram illustrating a fifth example of a calculation of tilt. Fig. 15 is a flowchart showing an example of a procedure for processing operations by an information processing device of a virtual cycling system according to an embodiment of the present invention. Fig. 16 is a flowchart showing an example of a procedure for user instruction determination by an information processing device of a virtual cycling system according to an embodiment of the present invention. Fig. 17 is a flowchart showing an example of a procedure for calibration by an information processing device of a virtual cycling system according to an embodiment of the present invention. Fig. 18 is a flowchart showing an example of a procedure for speed control by an information processing device of a virtual cycling system according to an embodiment of the present invention. Fig. 19 is a flowchart showing an example of a procedure for acquiring elapsed time and one foot position by an information processing device of a virtual cycling system according to an embodiment of the present invention. Fig. 20 is a flowchart showing an example of a procedure for reflecting the position for each pedal radius in an image by an information processing device of a virtual cycling system according to an embodiment of the present invention.Fig. 21 is a flowchart showing an example of the procedure for direction control processing by an information processing device of a virtual cycling system according to an embodiment of the present invention. Fig. 22 is a diagram showing an example of application of a virtual cycling system according to an embodiment of the present invention. Fig. 23 is a diagram showing a first example of head position detection related to direction control. Fig. 24 is a diagram showing a second example of head position detection related to direction control. Fig. 25 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention.

[0010] An embodiment of the present invention will now be described. FIG. 1 is a diagram showing an application example of a virtual cycling system according to an embodiment of the present invention. In this embodiment, there is no need to prepare professional or specialized equipment. Instead, by using a few general-purpose items, such as a regular camera and a spin bike similar to those used for aerobic exercise at a sports gym or at home, a user can operate a bicycle in a virtual space and participate in a competition using a low-cost system. The spin bike may be an exercise bike.

[0011] In the virtual cycling system shown in Fig. 1, a system consisting of an information processing device (reference symbol a in Fig. 1), a spin bike, a camera, and a monitor (reference symbol b in Fig. 1) captures a frontal image of a user riding the spin bike. The camera may be, for example, a camera mounted on a smartphone.

[0012] The information processing device analyzes the captured camera image (reference symbol c in FIG. 1) by estimating the skeleton and using markers, and identifies the positions of both shoulders and one foot of the user, in this case the right foot.

[0013] The information processing device determines the direction of travel of the spin bike from the identified shoulder positions at each time, determines the speed of travel of the spin bike from the identified right foot position, controls the bike in a virtual space (VR space), and reflects the position and movement of the bike in the VR space.

[0014] The information processing device also projects the player's viewpoint in the VR space onto a monitor in front of the spin bike.

[0015] FIG. 2 is a diagram showing an example of the positions of both shoulders and one foot of a user. In this embodiment, multiple elements of the user are read from a single camera image and the behavior of the bicycle in VR space, specifically, the direction and speed of travel, is controlled. To control the direction of travel, the information processing device detects the positions of the user's shoulders, in this case, "(x1, y1)" and "(x2, y2)," calculates the slope a of the two points using the linear function "y = ax + b," and converts this slope to θ according to the individual user's slope limit (symbol a in FIG. 2). The information processing device then converts the calculated slope into bicycle handlebar operation to determine the direction of travel of the bicycle (symbol b in FIG. 2).

[0016] To control the speed of travel, the information processing device detects the position of one of the user's feet, in this case "(xa, ya)", and calculates the time required for the user to pedal each pedal radius from the difference between the time when this position reaches a position near the top of the bicycle wheel pedal's possible position (symbol c in Figure 2) and the time when the detected position reaches a position near the bottom of the bicycle pedal's possible position (symbol d in Figure 2), and converts this time into the speed of the bicycle in VR space.

[0017] The control in this embodiment allows the user to operate the bicycle in the VR space with movements similar to those of a real bicycle in the physical world. Also, in this embodiment, the user can start the race and control the direction of the bicycle in the VR space using only information from the camera image, without operating the monitor screen or controller.

[0018] Furthermore, in this embodiment, the bicycle speed in the VR space can be controlled using only information from the camera image. The system according to this embodiment does not require the use of special and expensive parts, and can realize a usage environment using inexpensive and easily available items.

[0019] 3 is a block diagram showing an example of the functional configuration of an information processing device for a virtual cycling system according to one embodiment of the present invention. As shown in FIG. 3, information processing device 10 for a virtual cycling system according to one embodiment of the present invention includes an image acquisition unit 10-1, a skeleton (marker) detection unit 10-2, a user instruction determination unit 10-3, a race start determination unit 10-4, a calibration unit 10-5, a pedal speed calculation unit 10-6, a VR space velocity calculation unit 10-7, a tilt calculation unit 10-8, a VR space direction calculation unit 10-9, a VR space generation unit 10-10, and an output unit 10-11.

[0020] The image acquisition unit 10-1 acquires camera images captured by an external camera. (STEP 1) The user instruction determination unit 10-3 acquires the user's right hand position and head position from the skeleton (marker) detection unit 10-2, and performs user instruction determination to determine whether the acquired right hand position is higher than the acquired head position.

[0021] At the first determination, the user instruction determination unit 10-3 sends a start instruction to the calibration unit 10-5. After receiving a response from the calibration unit 10-5, the user instruction determination unit 10-3 resumes acquiring the above-mentioned position. At the second determination, the user instruction determination unit 10-3 sends a start instruction to the competition start determination unit 10-4.

[0022] (STEP 2) The skeleton (marker) detection unit 10-2 detects the coordinates of the user's hand position, head position, both shoulder positions, and one foot position from the camera image acquired by the image acquisition unit 10-1 by image analysis using techniques such as skeleton estimation and marker detection.

[0023] The skeleton (marker) detection unit 10-2 transmits the coordinates of the position of one hand and the position of the head to the user instruction determination unit 10-3. The skeleton (marker) detection unit 10-2 transmits the coordinate of the position of one foot to the calibration unit 10-5 and the pedal speed calculation unit 10-6. The skeleton (marker) detection unit 10-2 transmits the coordinates of the positions of both shoulders to the calibration unit 10-5 and the tilt calculation unit 10-8.

[0024] (STEP 3) (STEP 3-1) [Speed] The calibration unit 10-5 records the y-coordinate of one foot position for an arbitrary specified number of seconds. After the specified number of seconds has elapsed, the calibration unit 10-5 sorts the y-coordinate, which is a series of numerical values, in ascending order, and passes them to the competition start determination unit 10-4 as y_max, the lower limit of the numerical values ​​belonging to a predetermined percentage at the top, and y_min, the upper limit of the numerical values ​​belonging to a predetermined percentage at the bottom. In this embodiment, y_max is treated as the height near the top of the bicycle pedal that can be positioned by the user's driving, and y_min is treated as the height near the bottom of the bicycle pedal that can be positioned by the user's driving.

[0025] (STEP 3-2) [Direction] The calibration unit 10-5 instructs the player riding the spin bike to tilt their body (here, their shoulders) to the right as far as possible, and sets this tilt as a_max. The calibration unit 10-5 sets the handle angle coefficient "90° / a_max" and passes it to the competition start determination unit 10-4.

[0026] (STEP 4) The competition start determination unit 10-4 receives an instruction from the user instruction determination unit 10-3 and starts processing. The competition start determination unit 10-4 sends a start instruction together with "y_min / y_max" to the pedal speed calculation unit 10-6. The competition start determination unit 10-4 sends a_max and a start instruction to the slope calculation unit 10-8.

[0027] (STEP 5) In response to an instruction from the competition start determination unit 10-4, the pedal speed calculation unit 10-6 measures the time required for the user to pedal half a revolution, and passes the measurement result value to the VR spatial velocity calculation unit 10-7.

[0028] (STEP 6) The VR space velocity calculation unit 10-7 converts the value from the pedal velocity calculation unit 10-6 into bicycle velocity in the VR space, and passes this bicycle velocity to the VR space generation unit 10-10.

[0029] (STEP 7) In response to an instruction from the competition start determination unit 10-4, the gradient calculation unit 10-8 calculates the gradient a of the line segment connecting the two points at the user's shoulder positions. The gradient calculation unit 10-8 passes this gradient a to the VR space direction calculation unit 10-9.

[0030] (STEP 8) The VR space direction calculation unit 10-9 calculates θ using "θ = a * (90° / a_max)". If θ is -90° or less, it outputs -90° as the final θ, and if θ is 90° or more, it outputs 90° as the final θ. The VR space direction calculation unit 10-9 passes θ to the VR space generation unit 10-10 as the direction of travel of the bicycle in the VR space.

[0031] (STEP 9) Upon receiving the bicycle speed from the VR space speed calculation unit 10-7 and the bicycle's direction of travel in the VR space from the VR space direction calculation unit 10-9, the VR space generation unit 10-10 generates an image in the VR space that reflects these speed and direction, and passes it to the output unit 10-11. The output unit 10-11 displays the output image on an external monitor.

[0032] Next, the details of the speed-related processing performed by the calibration unit 10-5 will be described. Fig. 4 is a diagram showing an example of the recording results of the position of one of the user's feet. The user rotates the pedals of the spin bike for a predetermined number of seconds, and as shown in Fig. 4, the calibration unit 10-5 records the y-coordinate of the position of the user's right foot during this rotation.

[0033] Theoretically, the calibration unit 10-5 can acquire coordinate data up to an upper limit of "any number of seconds x camera fps." In this embodiment, it is assumed that the acquired y-coordinate data changes roughly like a sine wave when converted into a time series.

[0034] Fig. 5 is a diagram showing an example of the result of rearranging the recorded results of the position of one foot of the user. The calibration unit 10-5 rearranges the y-coordinate data recorded as described above in ascending order, and sets the lower limit value of the coordinates that correspond to a predetermined few percent of the top of the rearranged data as y_max shown in Fig. 5. Similarly, the calibration unit 10-5 sets the upper limit value of the coordinates that correspond to a predetermined few percent of the bottom of the rearranged data as y_min shown in Fig. 5.

[0035] Next, the details of the speed calculation by the pedal speed calculation unit 10-6 will be explained. Figure 6 is a diagram showing a first example of the recording results of the position of one of the user's feet, which is used to calculate the bicycle speed. The position of one of the user's feet immediately after the start of a race is not necessarily above y_max or below y_min, and the position is not determined.

[0036] Therefore, as shown in Figure 6, when the position of one foot of the user immediately after the start of the competition is below y_max, the pedal speed calculation unit 10-6 calculates the speed for each pedal radius from the time when the position of one foot of the user after the start of the competition first exceeds y_max.

[0037] The pedal velocity calculation unit 10-6 records in t1 the time "s1" when the position of one foot of the user first exceeds y_max (symbol a in FIG. 6) after the start of the race, and records in t2 the time "s2" when the position of one foot of the user first falls below y_min (symbol b in FIG. 6) after this recording.

[0038] The time it takes for the user to pedal half a revolution, i.e., the distance from the upper limit of the pedal position to the lower limit of the pedal position (symbol a1 in Figure 6), is "t2 - t1," or "s2 - s1." Using the gear ratio X and tire radius r, the speed when the user pedals half a revolution can be expressed as "Xπr / t2 - t1."

[0039] The pedal velocity calculation unit 10-6 passes the velocity when the user pedals half a revolution to the VR space generation unit 10-10, and in order to calculate the time required for the user to pedal the next half revolution of the pedal, i.e., the distance from the lower limit of the pedal position to the upper limit of the pedal position, first records in t1 the time when the position of one of the user's feet exceeds y_max for the first time since the time s2 was recorded (symbol c in Figure 6).

[0040] The time it takes the user to pedal half a revolution, i.e., the distance from the lowest pedal position to the highest pedal position, is t1 - t2, or pedal position s3 - s2. From the gear ratio X and tire radius r, the speed at which the user pedals half a revolution, i.e., the distance from the lowest pedal position to the highest pedal position, is expressed as "Xπr / t1 - t2." Thereafter, the bicycle speed is calculated from the time required for each pedal radius, and the results of this calculation are continuously passed to the VR space generation unit 10-10 until the bicycle in the VR space reaches the finish line.

[0041] Furthermore, in order to unify the equation for calculating the speed when the half pedal rotation is from the upper limit pedal position to the lower limit pedal position and when the half pedal rotation is from the lower limit pedal position to the upper limit pedal position, the pedal speed calculation unit 10-6 can calculate the speed using the formula "Xπr / |t2-t1|" using absolute values.

[0042] 7 is a diagram showing a second example of the recording result of the position of one of the user's feet, which is used to calculate the bicycle speed. As shown in FIG. 7, when the position of one of the user's feet is above y_max immediately after the start of the race, the pedal speed calculation unit 10-6 records the first detection time at t1, and thereafter performs the same processing as that performed when the position of one of the user's feet falls below y_min for the first time after the start of the race.

[0043] Next, the details of the processing related to direction by the calibration unit 10-5 will be described. Fig. 8 is a diagram illustrating an example of calculation of the inclination of the user's shoulders. Fig. 9 is a diagram illustrating an example of changes in the viewpoint and field of view in the VR space.

[0044] In this embodiment, as shown in (a) of Figure 8, after displaying an instruction on the screen to instruct the user to tilt their shoulders as far to the right as possible, the calibration unit 10-5 waits for an arbitrary number of seconds that is preset as the user's preparation time.

[0045] After an arbitrary number of seconds has elapsed, the calibration unit 10-5 determines that the user is tilting their shoulders to the maximum extent, and acquires the position coordinates (x1, y1) and (x2, y2) of the right shoulder and the left shoulder at this time.

[0046] The calibration unit 10-5 calculates the inclination of the line segment connecting the user's shoulders, "(y2-y1) / (x2-x1)," from the acquired coordinates (x1, y1) and (x2, y2), and sets this inclination as a_max. The state in which the user's shoulders are tilted to the maximum, in terms of steering operation, is the state in which the steering wheel is bent by 90° (FIG. 8(c)), with the state in which the user is facing forward (FIG. 8(b)) being 0°.

[0047] The calibration unit 10-5 converts the inclination a of both shoulders obtained during the race into a direction of travel and a steering angle in the VR space by calculating a steering coefficient by multiplying it by the steering coefficient "90° / a_max."

[0048] The handle coefficient is expressed as "(a*90°) / a_max[°]". When converting the angle with this handle coefficient to rad, the handle coefficient becomes "2a / (π・a_max)".

[0049] For example, if the user's maximum tilt is 3, and the tilt of the line segment at the position of both shoulders when the user tilts their shoulders during a match is "1," then the real-life tilt of "3" will correspond to an angle of 90° in the VR space (see Figure 9).

[0050] The user's real-life tilt of "1" during the race is multiplied by the steering coefficient "90° / 3" to become 30°, which means that the direction of travel is tilted 30° to the right in VR space. When θ becomes a negative value, the front is considered to be 0° in VR space and the steering is tilted to the negative side, that is, to the left.

[0051] Next, a specific example of the calculation of the tilt by the tilt calculation unit 10-8 will be described. (1) When the shoulders are not tilted Figure 10 is a diagram illustrating a first example of the calculation of the tilt. When the user is not tilting their shoulders (Figure 10(a)), the tilt of the line segment calculated by the tilt calculation unit 10-8 from the position of both shoulders of the user is 0. At this time, there is no change in the direction of the handle in the VR space (Figure 10(b)), and the field of view in the VR space does not change.

[0052] (2) When the shoulders are tilted to the right Figure 11 is a diagram illustrating a second example of tilt calculation. When the user tilts their shoulders to the right (Figure 11(a)), the tilt of the line segment calculated by the tilt calculation unit 10-8 from the position of the user's shoulders is a positive number. In this case, the range of possible values ​​for θ (Figure 11(b)) converted from the tilt, i.e., the range of direction change in VR space, is "0° to 90°." In this case, the handlebars of the bicycle are turned to the right by θ° in VR space, and the field of view in VR space shifts to the right.

[0053] (3) When the shoulders are tilted to the left Figure 12 is a diagram illustrating a third example of tilt calculation. When the user tilts their shoulders to the left (Figure 12(a)), the tilt of the line segment calculated by the tilt calculation unit 10-8 from the positions of the user's shoulders is a negative number. In this case, the range of possible values ​​for θ (Figure 12(b)) converted from the tilt, i.e., the range of direction change in VR space, is "-90 to 0°." In this case, the steering wheel is turned to the left by θ° in VR space, and the field of view in VR space shifts to the left.

[0054] (4) When the shoulders are tilted more to the right than at the time of calibration: Figure 13 is a diagram illustrating a fourth example of tilt calculation. When the user tilts their shoulders more to the right (symbol b in Figure 13) than at the time of calibration (symbol a in Figure 13), the tilt of the line segment calculated by the tilt calculation unit 10-8 from the positions of the user's shoulders is a positive number. The value obtained by multiplying this tilt by the steering wheel tilt coefficient is 90° or greater. In this case, the angle θ converted from the tilt is set to 90°, and the field of view in the VR space is tilted 90° to the right.

[0055] (5) When the shoulders are tilted more to the left than at the time of calibration: FIG. 14 is a diagram illustrating a fifth example of tilt calculation. When the user tilts their shoulders more to the left (symbol b in FIG. 14) than at the time of calibration (symbol a in FIG. 14), the tilt of the line segment calculated by the tilt calculation unit 10-8 from the positions of the user's shoulders is a negative number. The value obtained by multiplying this tilt by the steering wheel tilt coefficient is −90° or less. In this case, the angle θ converted from the tilt is set to −90°, and the field of view in the VR space is tilted 90° to the left.

[0056] 15 is a flowchart showing an example of the procedure of processing operations by the information processing device of the virtual cycling system according to one embodiment of the present invention. First, the user instruction determination unit 10-3 performs a first user instruction determination process (S10). The calibration unit 10-5 performs calibration (S20). Next, the user instruction determination unit 10-3 performs a second user instruction determination process (S30).

[0057] After S30, the pedal velocity calculation unit 10-6, the VR space velocity calculation unit 10-7, and the VR space generation unit 10-10 perform velocity control processing (S40).After S30, the tilt calculation unit 10-8, the VR space direction calculation unit 10-9, and the VR space generation unit 10-10 perform direction control processing (S50).

[0058] Figure 16 is a flowchart showing an example of the steps of the user instruction determination process performed by the information processing device of the virtual cycling system according to one embodiment of the present invention. Here, the details of the processes of S10 and S30 shown in Figure 15 are explained. The user instruction determination unit 10-3 inputs camera images (S10-1). The user instruction determination unit 10-3 detects the position of one of the user's hands and the position of the user's head from the camera images (S10-2). The following explanation assumes that the position of the user's right hand has been detected, but the position of the user's left hand may also be detected.

[0059] The user instruction determination unit 10-3 determines whether the user's right hand is positioned above the head (S10-3). If the answer is "no" in S10-3, the process returns to S10-1. If the answer is "yes" in S10-3, the user instruction determination unit 10-3 determines this state as the result of the user instruction determination process (S10-4).

[0060] FIG. 17 is a flowchart showing an example of the calibration procedure performed by the information processing device of the virtual cycling system according to one embodiment of the present invention. Here, the process of S20 shown in FIG. 15 will be described in detail. The calibration unit 10-5 displays an instruction on the monitor screen to instruct the user to rotate the pedals of the spin bike (S20-1). The calibration unit 10-5 inputs camera images (S20-2) and detects the position of one of the user's feet from the camera images (S20-3). The calibration unit 10-5 records the height coordinate in y(i) (S20-4). i is obtained from the number of seconds since the start of calibration.

[0061] If the arbitrary number of seconds has not elapsed since the start of calibration (no in S20-5), the process returns to S20-2. If the arbitrary number of seconds has elapsed since the start of calibration (yes in S20-5), the calibration unit 10-5 sets the upper X% of y(i), for example the lower limit of the range of a few percent, as y_max, and sets the lower X% of y(i), for example the upper limit of the range of a few percent, as y_min (S20-6).

[0062] The calibration unit 10-5 displays on the monitor screen an instruction to the user to tilt their shoulders to the right as far as possible (S20-7). When a certain number of seconds has elapsed since the instruction was displayed in S20-7 (yes in S20-8), the calibration unit 10-5 inputs camera images (S20-9) and detects the positions of the user's shoulders from the camera images (S20-10). The calibration unit 10-5 calculates a line segment from the coordinates of the detected positions of the shoulders, calculates the slope, and sets this as a_max (S20-11).

[0063] The calibration unit 10-5 calculates the handle angle coefficient K by "K = 90° / a_max" (S20-12). The calibration unit 10-5 notifies the user on the monitor that the race is ready to begin (S20-13).

[0064] 18 is a flowchart showing an example of the speed control process performed by the information processing device of the virtual cycling system according to one embodiment of the present invention. Here, the details of the process of S40 shown in FIG. 15 are explained. The pedal speed calculation unit 10-6 acquires the elapsed time T from the start of the race and the coordinate y(T) of the position of one of the user's feet (S40-1).

[0065] If y(T) does not exceed y_max (no in S40-2), S40-1 is repeated. If y(T) exceeds y_max (yes in S40-2), the pedal velocity calculation unit 10-6 records T in t1 (S40-3).

[0066] Next, the pedal velocity calculation unit 10-6 acquires the elapsed time T from the start of the race and the coordinate y(T) of the position of one of the user's feet (S40-4). If y(T) is not below y_min (no in S40-5), S40-4 is repeated. If y(T) is below y_min (yes in S40-5), the pedal velocity calculation unit 10-6 records T at t2 (S40-6). The pedal velocity calculation unit 10-6, VR space velocity calculation unit 10-7, and VR space generation unit 10-10 reflect the speed for each pedal radius of the bicycle in the camera image (S40-7).

[0067] If the user has reached the goal of the spin bike race (yes in S40-8), the series of processes in S40 ends. If the user has not reached the goal of the spin bike race (no in S40-8), the pedal speed calculation unit 10-6 acquires the elapsed time T from the start of the race and the coordinate y(T) of the position of one of the user's feet (S40-9).

[0068] If y(T) does not exceed y_max (no in S40-10), S40-9 is repeated. If y(T) exceeds y_max (yes in S40-10), the pedal velocity calculation unit 10-6 records T in t1 (S40-11). The pedal velocity calculation unit 10-6, VR space velocity calculation unit 10-7, and VR space generation unit 10-10 reflect the speed for each pedal radius of the bicycle in the camera image (S40-12).

[0069] If the user has not reached the goal of the spin bike race (no in S40-12), the process returns to S40-4. If the user has reached the goal of the spin bike race (yes in S40-12), the series of processes in S40 ends.

[0070] 19 is a flowchart showing an example of the processing procedure for acquiring elapsed time and one foot position by an information processing device of a virtual cycling system according to an embodiment of the present invention. Here, the processing of S40-1, S40-4, and S40-9 shown in FIG. 18 will be described in detail.

[0071] First, the pedal velocity calculation unit 10-6 inputs the camera image (S40-1-1) and records the elapsed time [s] from the start of the race at time T (S40-1-2). The pedal velocity calculation unit 10-6 detects the position of one of the user's feet from the camera image (S40-1-3) and records the height coordinate of the position of the user's one foot in y(T) (S40-1-4).

[0072] 20 is a flowchart showing an example of the processing procedure for reflecting the position of each pedal radius in the video by the information processing device of the virtual cycling system according to one embodiment of the present invention. Here, the details of the processing of S40-7 and S40-12 shown in FIG. 18 are described.

[0073] First, the pedal speed calculation unit 10-6 calculates the time (time) required for the user to pedal the bicycle using the equation "time = |t2 - t2|" (S40-7-1). The VR space speed calculation unit 10-7 converts the time (time) into the bicycle speed in the XR (cross reality) sports space using the equation "bicycle speed [mm / s] = (gear ratio × tire radius [mm] × π) / time" (S40-7-2). The VR space generation unit 10-10 outputs an image that adapts this bicycle speed to the player's speed in the XR space (S40-7-3).

[0074] Figure 21 is a flowchart showing an example of the procedure for direction control processing by the information processing device of the virtual cycling system according to one embodiment of the present invention. Here, the details of the processing of S50 shown in Figure 15 are explained. The tilt calculation unit 10-8 inputs camera images (S50-1) and detects the positions of the user's shoulders from the camera images (S50-2). The tilt calculation unit 10-8 calculates a line segment from the coordinates of the positions of the user's shoulders and calculates the tilt a (S50-3).

[0075] The VR space direction calculation unit 10-9 calculates the angle θ by "θ = a × K" (S50-4). If this θ is less than or equal to -90° (yes in S50-5), the VR space direction calculation unit 10-9 sets θ to -90° (S50-6).

[0076] If the answer to S50-5 is "no" and the angle θ calculated in S50-4 is 90° or greater (yes in S50-7), the VR space direction calculation unit 10-9 sets the angle θ to 90° (S50-8).

[0077] If S50-7 is "no," the VR space direction calculation unit 10-9 outputs the θ calculated in S50-4 to the VR space generation unit 10-10. After the setting in S50-6 or the setting in S50-8, the VR space direction calculation unit 10-9 outputs the set θ to the VR space generation unit 10-10. The VR space generation unit 10-10 outputs an image when the traveling direction in the XR space is changed to the θ output from the VR space direction calculation unit 10-9 (S50-9).

[0078] Next, application examples of this embodiment will be described. As a first application example, "implementation without a spin bike" will be described. In this first application example, when a spin bike is not available or when implementing the present invention more simply, speed control can be substituted by the movement of the user's feet.

[0079] In addition, the tilt of the user's shoulders is used for directional control, as in the above-described embodiment. Also, even if the user holds a stick or the like with both hands to make the movement smoother, it is possible to perform the movement as long as the position of the user's shoulders can be determined.

[0080] Fig. 22 is a diagram showing an application example of a virtual cycling system according to an embodiment of the present invention, in which a user sitting in a chair is photographed from the front by the camera in a system consisting of an information processing device (reference symbol a in Fig. 22), a camera, and a monitor (reference symbol b in Fig. 22).

[0081] The information processing device analyzes the captured camera image (reference symbol c in Figure 22) by estimating the user's skeleton and using markers to identify the positions of the user's shoulders and one foot. The subsequent processing is the same as that performed by the virtual cycling system shown in Figure 1.

[0082] The user's foot movement envisioned here includes a foot rotation motion similar to the user pedaling a spin bike, but it may also be a motion in which both of the user's feet simply move up and down, as shown in Figure 22. It may also be a motion in which the user's toes move up and down.

[0083] As a second application example, "changing the detection position" will be described. Fig. 23 is a diagram showing a first example of head position detection related to directional control. Fig. 24 is a diagram showing a second example of head position detection related to directional control.

[0084] In this second application example, the head position and right hand position in the user instruction determination unit 10-3 are determined based on a fixed standard, and the latter can be adjusted up or down as desired by the user, and the head may be replaced with the nose, mouth, shoulders, neck, etc., and the right hand position may be replaced with the right elbow or left hand. Speed ​​control is not limited to the user's toes, and the knee position can also be used as an alternative.

[0085] The direction control is not limited to the positions of both shoulders, but can also be substituted by the positions of both eyes, both ears, or both hands. In the second application example, the user's imaged parts may be any parts that form a pair when the user's upper body is imaged from the front. The direction control can be performed based on only one part, such as the head, instead of two parts, and can also be substituted by the position of the nose or mouth.

[0086] During calibration, the head position can be obtained from images taken at two points in time: when the user is tilted to the maximum and when facing forward. Also, rather than obtaining two points from images taken at the same time, two points can be obtained from two images taken at different times to determine the tilt a_max. Along with a_max, the coordinates (x2, y2) when facing forward are also saved to calculate the tilt each time during the competition.

[0087] The two coordinates used to calculate the tilt a during a match are the head position coordinates (x1, y1) (see Figure 23) and (x2, y2) (see Figure 24). The tilt a can be calculated using the formula "a = (y2 - y1) / (x2 - x1)".

[0088] The processing in the second application example can be realized by replacing (STEP 2), (STEP 3-2), (STEP 4), and (STEP 7) of (STEP 1) to (STEP 9) in the above-described embodiment with the following:

[0089] (STEP 2) The skeleton (marker) detection unit 10-2 detects the coordinates of the user's hand, head, shoulders, and foot positions by image analysis using techniques such as skeleton estimation or marker detection from the camera image. The coordinates of the head position are sent to the calibration unit 10-5 and the tilt calculation unit 10-8.

[0090] (STEP 3-2) [Direction] The calibration unit 10-5 measures the coordinates (x1, y1) of the head position when the user's head is tilted to the right to the maximum extent.

[0091] The calibration unit 10-5 measures the coordinates (x2, y2) of the head position when the user faces forward. The calibration unit 10-5 sets the slope of the line segment connecting the measured (x1, y1) and (x2, y2) as a_max. The calibration unit 10-5 sets a handle angle coefficient (90° / a_max) and passes this coefficient along with (x2, y2) to the competition start determination unit 10-4.

[0092] (STEP 4) The competition start determination unit 10-4 sends a start command to the gradient calculation unit 10-8 together with a_max and the coordinates (x2, y2).

[0093] (STEP 7) In response to an instruction from the competition start determination unit 10-4, the tilt calculation unit 10-8 calculates the tilt a of the line segment connecting the coordinates (x1, y1) of the head position at that time and the coordinates (x2, y2) acquired during calibration. The tilt calculation unit 10-8 passes this tilt a to the VR space direction calculation unit 10-9.

[0094] Fig. 25 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention. In the example shown in Fig. 25, the information processing device 10 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0095] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0096] An input device 500 and an output device 600 attached to the information processing device 10 and used by a user or the like are connected to the input / output interface 113. The input / output interface 113 can take in operation data input by a user or the like through the input device 500 such as a keyboard, a touch panel, a touchpad, or the like, and can output and display output data to an output device 600 including a display device using a liquid crystal or an organic electroluminescence (EL) display, or the like. The input device 500 and the output device 600 may be devices built into the information processing device 10, or may be input devices and output devices of other information terminals that can communicate with the information processing device 10 via a network.

[0097] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and can store programs necessary to execute various control processes, etc., according to one embodiment.

[0098] The data memory 112 is a tangible storage medium that is, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during various processes.

[0099] An information processing apparatus 10 according to one embodiment of the present invention can be configured as an information processing apparatus having the units shown in FIG. 1 as software-based processing function units.

[0100] The information storage unit used as a work memory or the like by each unit of the information processing device 10 can be configured by using the data memory 112 shown in Fig. 25. However, these configured storage areas are not essential components within the information processing device 10, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0101] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0102] The methods described in each embodiment can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), or semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables or data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0103] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0104] 10... Information processing device 10-1... Image acquisition unit 10-2... Skeleton (marker) detection unit 10-3... User instruction determination unit 10-4... Competition start determination unit 10-5... Calibration unit 10-6... Pedal speed calculation unit 10-7... VR space speed calculation unit 10-8... Inclination calculation unit 10-9... VR space direction calculation unit 10-10... VR space generation unit 10-11... Output unit

Claims

1. An information processing device comprising: a detection unit that detects the positions of both shoulders and one foot of a user using captured images of the user riding a bicycle; a speed calculation unit that calculates the traveling speed of the bicycle being ridden by the user in a virtual space based on the position of one foot of the user at a certain time detected by the detection unit; a direction calculation unit that calculates the slope of a line segment connecting the positions of both shoulders of the user detected by the detection unit and calculates the traveling direction of the bicycle being ridden by the user in the virtual space based on the calculated slope; and a generation unit that generates an image of the bicycle being ridden by the user in the virtual space traveling at the traveling speed calculated by the speed calculation unit and in the traveling direction calculated by the direction calculation unit.

2. The information processing device described in claim 1, wherein the detection unit detects the height position of one of the user's feet at each of a plurality of times within the fixed period of time, and the speed calculation unit calculates the time required for the user to pedal half a revolution of the bicycle wheel based on the detection results of the detection unit's detection of the height position of one of the user's feet at each of a plurality of times within the fixed period of time, and calculates the traveling speed of the bicycle based on the calculated time, the gear ratio of the bicycle wheel, and the radius of the wheel.

3. The information processing device described in claim 1, wherein the direction calculation unit calculates the angle of the handlebar operation of the bicycle driven by the user in the virtual space based on the calculated inclination and the inclination when the user driving the bicycle leans their body to the maximum, and calculates the calculated angle as the direction of travel of the bicycle driven by the user in the virtual space.

4. The information processing device described in claim 2, wherein the speed calculation unit calculates the time required for the user to pedal the bicycle wheel halfway around based on the difference between the time when the detection unit's detection result of the height position of one of the user's feet at each of multiple times during the fixed period of time reaches a first position, which is a position near the top of the pedal, and the time when the detection result of the user's height position of one of the user's feet reaches a second position, which is a position near the bottom of the pedal.

5. A method performed by an information processing device, comprising: a detection unit of the information processing device detecting the positions of both shoulders and one of the user's feet using images captured while the user is riding a bicycle; a speed calculation unit of the information processing device calculating the traveling speed of the bicycle being ridden by the user in a virtual space based on the position of one foot of the user at a certain time detected by the detection unit; a direction calculation unit of the information processing device calculating the slope of a line segment connecting the positions of both shoulders of the user detected by the detection unit, and calculating the traveling direction of the bicycle being ridden by the user in the virtual space based on the calculated slope; and a generation unit of the information processing device generating an image of the bicycle being ridden by the user in the virtual space traveling at the traveling speed calculated by the speed calculation unit and in the traveling direction calculated by the direction calculation unit.

6. An information processing program that causes a processor to function as each part of the information processing device according to any one of claims 1 to 4.

Images