Augmented reality doll

Abstract

A system and method for enhancing the visual expression of a visual object in a seamless and vivid manner, the visual object having a different overlay visual image therewith, comprising means for providing information about the position and movement of each movable part of the object; image acquisition means for acquiring images and videos of the object; processing means for calculating the three-dimensional distance and position of the movable parts and the object; image processing means for generating an overlay image at the location of the object; means for displaying the overlay image. The system itself is housed in a smartphone-based virtual reality headset.

Description

Technical Field

[0001] The present application relates to a system and method for enhancing or changing, switching the visual expression and perception of a person or inanimate object with different visualization effects in a seamless and vivid manner. More specifically, the present application employs image and video capture, processing and display, and virtual reality representation. Background Art

[0002] Virtual reality (VR) and augmented reality (AR) offer users the possibility to visually immerse themselves in another virtual world. However, they are limited in that they only provide visual stimulation.

[0003] Based on this, the inventor proposed the present application to connect the virtual visual world to the physical tactile environment close to them. Summary of the invention

[0004] According to a first aspect of the present application, a system for enhancing the visual expression of an object in a seamless and vivid manner is provided, wherein the object has a different overlay visual image, and the object is a toy or a tights, and the tights are wearable by a human body.

[0005] The system comprises:

[0006] a camera for acquiring images and videos of the subject;

[0007] a rotary encoder embedded in each joint of each movable part of the object, for acquiring the angle of each movable part;

[0008] wherein the rotary encoder is connected to a microcontroller via wires, and the microcontroller sends data from each joint to a computer or smartphone;

[0009] an inertial measurement unit for position tracking, the position information being sent to the smartphone;

[0010] Position processing means for calculating the posture of the entire object through the skeleton data and the position data;

[0011] image processing means for generating a computer generated overlay image at the location of the object; and

[0012] a display device for displaying the overlay image,

[0013] Wherein, the position processing device, the image processing device, the display device and the camera are all parts of a virtual reality headset based on a smartphone worn by a user;

[0014] A smart device application APP installed on the smart phone;

[0015] means for calculating the position and posture of said object and overlaying a computer generated overlay image relative to the user;

[0016] The computer-generated overlay image is closely aligned with the object so that a user can feel a sensation of touching the object when controlling the object.

[0017] Another embodiment of the present application provides a method for enhancing the visual expression of a human body or an inanimate object in a seamless and vivid manner, wherein the human body or inanimate object has an overlaid visual image that is different therefrom, and the method includes the above-mentioned system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present application will now be described by way of example only with reference to the accompanying drawings, in which:

[0019] Figure 1 A data flow diagram of a system using visual identifiers according to an embodiment of the present application is shown;

[0020] Figure 2 A data flow diagram of a system using an inertial measurement unit (IMU) according to an embodiment of the present application is shown;

[0021] Figure 3 A data flow diagram of a system using a rotary encoder according to an embodiment of the present application is shown; and

[0022] Figure 4 Depicted is the use of a chroma key bodysuit in this application. DETAILED DESCRIPTION

[0023] The following are examples and methods for implementing the present application:

[0024] 1. Children's toys

[0025] The present application proposes a toy, such as a doll, which is modified or has elements added to it so that a 3D computer-generated model with similar proportions or effects is superimposed in a virtual reality (VR) or augmented reality (AR) environment. The superimposed model is aligned very closely with the physical toy so that the user can feel what it is like to touch the superimposed model. Essentially, in the VR / AR environment, the space occupied by the superimposed model or effect is the same as the space occupied by the physical toy in the real world. Computer-generated effects can be attached to the toy, such as flames or sticky substances visible in VR / AR, without actually having any flames or sticky substances in the real world. An example of use in this regard is the virtual flames emitted from the tip of the gun of a standard "tank and soldier" toy that simulates shooting.

[0026] The first aspect of the present application is achieved by an optical marker. Figure 1 , each movable part of the toy is connected to various markers of different appearance or color, and the toy is, for example, a doll. The VR headset can be a smartphone VR headset with an embedded camera, through which the optical markers on the toy can be obtained. A smart device application (App) is installed on the smartphone (11). The smartphone application displays a computer-generated overlay model and uses the image obtained by the camera to perform calculations to place the computer-generated overlay model in the same space as the toy (12). Each movable part of the toy has a different marker. When the smartphone camera detects the marker on the loader representing the toy bulldozer, the smartphone application places the computer-generated dust and dust drop overlay model at the location of the loader of the physical toy so that the loader of the toy bulldozer looks like it is covered with dust. Positioning is achieved by calculating the angle of the marker and its distance from the camera. The position and posture of the toy are calculated, and a 3D representation of the toy is superimposed relative to the user (13).

[0027] This is done with each marker detected by the camera, which is used to position each movable part of the computer-generated overlay model. The distance and angle to the user are measured. For example, if the loader of a toy bulldozer is 2m away from the camera and at a 30 degree angle to the camera, then in the VR / AR space, the computer-generated overlay model is 2m away from the user and at a 30 degree angle to the user. The application also displays the computer-generated overlay model and performs this calculation.

[0028] The second aspect of the present application is achieved by an inertial measurement unit (IMU).

[0029] See also Figure 2 An inertial measurement unit (IMU) is embedded inside each movable part of the toy (20).

[0030] Each IMU will record the current angle and position of each movable part. This information is sent wirelessly to the application (21), such as via Wi-Fi or Bluetooth. The application will set the angle and position of the computer-generated superimposed model to correspond to the toy (22) and its movable parts.

[0031] The position and pose of the doll are calculated by the IMU and a 3D representation of the doll is superimposed relative to the user (23). For example, if the toy is 2m in front of the user, the computer-generated superimposed model will appear 2m in front of the user in the VR / AR space. The IMU requires a portable power source that is also embedded in the toy and has an access point for charging.

[0032] The third aspect of the present application is implemented through an encoder.

[0033] A rotary encoder is used to store the angle of each movable part of the toy. For the toy excavator arm, a rotary encoder is embedded in each joint of the excavator arm and stores the angle of each joint on a microcontroller (30). The rotary encoder is connected to the microcontroller via wires, which sends data from the movable joint to a computer or smartphone (31). The toy is also embedded with an IMU for position tracking, and a portable power source with a charging access point. The IMU sends the position information to a computer or smartphone, where an application uses the skeletal data and position data to calculate the posture of the entire toy (32) and displays a computer-generated superimposed model of the toy to the user (33) in a virtual reality / augmented reality space.

[0034] The system is open to accept a wide range of encoders and computing stations, or to connect the IMU.

[0035] A microcontroller / Arduino board might be used.

[0036] The fourth aspect of the present application is referred to as post-processing.

[0037] This method is useful for applications that make toys look like cartoons. The toy must physically resemble the target cartoon. Each movable part of the toy has a different color. For two parts that are directly connected, such as the loader and arm of the toy excavator, there is an additional color bar between the two parts (in this case, at the joint). The additional color bar is the same between every two directly touching parts. The application uses the smartphone's camera to detect two different colors that are adjacent to each other. If two different colors are adjacent to each other and neither is an "additional color bar", a black outline is drawn between them. All colors, excluding the background and outline, are replaced with the corresponding cartoon color.

[0038] This system serves two purposes.

[0039] 1: Add computer generated effects (such as fire) to the toy to make it look more realistic. For example, instead of a toy soldier having a plastic molded muzzle flash, the toy soldier can have a computer generated muzzle flash.

[0040] 2: Make toys more like the objects they are meant to represent with computer graphics. For example, a plastic toy soldier can be transformed into a more realistic-looking computer-generated soldier that blinks and breathes. In another example, a toy that resembles a cartoon character on television can more closely match the cartoon art style of the show when computer graphics are superimposed on it.

[0041] 2. Tights

[0042] Another embodiment of the present application proposes a tights product. The product has two main users: a "wearer" and a "viewer".

[0043] The silicone bodysuit is modified or features are added to overlay an anthropomorphic 3D computer-generated model in a virtual reality (VR) or augmented reality (AR) environment. The wearer wears the bodysuit while a viewer views the overlay model using a smartphone VR viewer. The overlay model is aligned very closely with the wearer of the bodysuit so that the viewer of the smartphone VR viewer can feel the sensation of touching the overlay model while the wearer controls the overlay model. Essentially, in the VR / AR environment, the space occupied by the overlay model is the same as the space occupied by the wearer in the real world.

[0044] Method 1: (Optical Marker Method) - The bodysuit has various markers of different appearance or color attached to it to represent various body parts. The VR headset needs to be a smartphone VR headset, which has a camera through which the optical markers on the bodysuit can be seen. The application needs to be installed on the smartphone. The smartphone application displays the superimposed model and uses the camera's acquisition content to calculate to place the superimposed model in the same space as the bodysuit. Each limb of the bodysuit has a different marker. When the smartphone's camera detects the marker representing the right arm, the smartphone's application places the superimposed model's right arm at the location of the doll's right arm. The positioning is achieved by calculating the angle of the marker and its distance from the camera. This is done by feeding each marker detected by the camera to the corresponding position. For example, if the right arm of the bodysuit is 2m away from the camera and at a 30-degree angle, the right arm of the superimposed model is 2m away from the viewer and at a 30-degree angle in the VR / AR space.

[0045] The application also displays the overlay model. Different types of markers with various appearances are possible. The application also performs calculations.

[0046] Method 2: (Post-Processing) - This method is most useful for making a cartoon representation of the wearer with simple colors and outlines. The wearer must be physically very similar to the target overlay model. Each limb of the doll has a different color. The different color parts include the left forearm, left upper arm, left hand, right forearm, right upper arm and right hand, left thigh, left calf, left foot, right thigh, right calf, right foot, head, neck, left chest, right chest, abdomen, chest, pelvic area and each phalanx of each finger and toe, which will all have different colors. If two parts are directly connected, such as the right forearm and right upper arm, there will be an additional color bar between them (in this case, the elbow). The additional color bar between each two parts is the same. The application uses the smartphone's camera to detect two different colors that are adjacent to each other. If two different colors are adjacent to each other and neither is an "additional color", a black outline is drawn between them. A cartoon-style black line is generated around the cartoon. All colors excluding the background and outlines are replaced with the cartoon skin color.

[0047] The silicone of the bodysuit may be colored prior to solid state molding thereof, or the silicone of the bodysuit may be colored after molding thereof.

[0048] Method 3: (Chroma Key Method) - This method is most useful for representing the wearer as a cartoon with simple colors and outlines. The bodysuit is colored with a solid "chroma key" color, and the viewer's smartphone will post-process the camera feed to replace that color with the cartoon skin color. The wearer of the bodysuit needs to be brightly illuminated by a light mounted on the viewer's VR headset. This will give the "chroma key" color an even illumination that will dim near the edges of the bodysuit. This dimming near the edges will create a blurred outline so that the viewer can make out the shape of the wearer, such as when the wearer's arms extend beyond their torso.

[0049] Figure 4 A similar example of a person wearing a "chroma key" bodysuit (40) is shown. The shadows around the fingers indicate the position of the hands in front of the wearer. Normally, the wearer of a chroma key bodysuit would evenly light all around so that all shadows are eliminated, but for this solution, lights are placed in front of the wearer (mounted on a VR headset) to create these shadows so that the viewer can still make out the wearer's shape in this case.

[0050] In most of these methods, there is also a way to determine the position of the doll's face. This can be done using visual markers. This will allow the face to have movable functions such as blinking superimposed on the doll. It may also be achieved using existing "face motion capture" technology.

Claims

1. A system for enhancing the visual expression of an object in a seamless and vivid manner, the object having a different overlay visual image, characterized in that: The object is a toy or a tights, and the tights are wearable by a human body. The system comprises: a camera for acquiring images and videos of the subject; a rotary encoder embedded in each joint of each movable part of the object, for acquiring the angle of each movable part; wherein the rotary encoder is connected to a microcontroller via wires, and the microcontroller sends data from each joint to a computer or smartphone; an inertial measurement unit for position tracking, wherein position information detected by the inertial measurement unit is sent to the smartphone; Position processing means for calculating the posture of the entire object using the skeleton data and the position data; image processing means for generating a computer-generated overlay image at the location of the object; and a display device for displaying the overlay image, The position processing device, the image processing device, the display device, and the camera are all parts of a smartphone-based virtual reality headset worn by a user; A smart device application APP installed on the smartphone; means for calculating the position and pose of said object and overlaying a computer-generated overlay image relative to the user; The computer-generated overlay image is closely aligned with the object so that a user can feel a sensation of touching the object when controlling the object.

2. A method for enhancing the visual expression of a human body or inanimate object in a seamless and vivid manner, wherein the human body or inanimate object has a different overlay visual image, characterized in that The method comprises the system of claim 1.

Images