Method, system and AR glasses for performing realistic social activities using virtual scenes

Abstract

To provide a method, system and AR glasses for performing realistic social activities using virtual scenes. SOLUTION: A method for performing a real social activity using a virtual scene includes a cloud service side and a live streaming device. The method includes six steps. A system for performing a real social activity using a virtual scene includes a cloud service side and a live streaming device, the cloud service side includes a cloud service processor, the live streaming device includes a terminal processor, the live streaming device further includes at least three sets of video cameras, AR glasses, inner-ear wireless earphones, a positioning device, and a gyroscope, the cloud service processor and the terminal processor are connected through TCP / IP communication, the video cameras, the VR head mounted display, the inner-ear wireless earphones, the positioning device, and the gyroscope are electrically connected to the terminal processor, the positioning device is fixed and integrally installed with the gyroscope, and the positioning device and the gyroscope are fixed to the chest of the streamer. The system also includes AR glasses.

Description

[Technical field]

[0001] The present invention relates to the technical field of virtual reality, and in particular to a method, system and AR glasses for performing realistic social activities using virtual scenes. [Background technology]

[0002] Virtual reality technology, or VR, includes computer, electronic information, and simulation technology, and the basic implementation method is that the computer simulates a virtual environment and gives people a sense of immersion in the virtual environment. Currently, with the development of VR technology, VR is being applied to various technical fields.

[0003] Patent Document 1 discloses a VR social system and method based on real-time 3D reconstruction of the human body. The disclosed patent document has the following shortcomings. First, the human body that is 3D reconstructed in the VR scene is not a real human body image that is live-streamed. Second, the patent document does not solve the problem of how to match the position change of the real human body with the virtual scene, and there is a problem that the direction and speed of the movement of the human body in reality do not match the direction and speed of the movement of the human body in the virtual scene. Third, due to the mismatch between the position change of the real human body and the virtual scene, it is not possible to accurately establish a connection with others through the virtual scene, and the social efficiency is low. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Chinese Patent No. 2017103756194 [Patent Document 2] China Patent No. 2017215701499 (Name of invention: Projection-type virtual reality VR glasses display device) [Patent Document 3] China Patent No. 2018105750619 (Name of invention: Method and device for automatic location detection and wireless charging of drones) Summary of the Invention [Problem to be solved by the invention]

[0005] To solve the above technical problems, the present invention provides a method and system for conducting realistic social activities using virtual scenes. [Means for solving the problem]

[0006] The specific technical measures are: A method for performing a realistic social activity using a virtual scene with a cloud service side and a live streaming device, the method comprising the steps of: Step 1: The cloud service side obtains the number N of live streaming rooms participating in the social virtual scene according to the number of live streaming devices, where the live streaming room is a live streaming scene in reality, and N≧2; Step 2: Create N+1 identical 3D coordinate systems based on the number N of live streaming rooms, and each live streaming room creates one 3D coordinate system. N live streaming rooms create 1 to N 3D coordinate systems. The cloud service creates the N+1th 3D coordinate system in the virtual scene. Step 3: Set the N+1 three-dimensional coordinate systems, each of which is composed of an x-axis, a y-axis, and a z-axis, and each of the x-axis, y-axis, and z-axis has the same length unit; The floor of the virtual scene is defined as a plane formed by the x-axis and y-axis of each 3D coordinate system, and the floor of the live streaming room is also a plane formed by the x-axis and y-axis of the 3D coordinate system. The spatial range occupied by the virtual scene in the N+1th 3D coordinate system is K N+1 The corresponding spatial range of each live streaming room in the 3D coordinate system is K 1 -K N and K 1 -K N There are no obstacles in K 1 -K NThe appearance of a real person in the live broadcast may be broadcast live in 3D, and the term "obstacle" refers to the degree of obstruction that prevents the completion of the live broadcast. Obstacles that can be filtered and corrected during the live broadcast are considered to be non-obstacles. Step 4: There is at least one live streaming room defined as having a streamer in each live streaming room, and the number of live streaming rooms defined as having a streamer in each live streaming room is M, where N≧M≧2. Each live streaming room in the M live streaming rooms collects the human body position information and human body appearance information of each streamer, and the streamer's voice information and voice position information, and transmits them to the cloud service side, and the cloud service side processes the appearance information of each streamer in each live streaming room into 3D avatar information; Step 5: Replace step 4, and there is at least one live streaming room defined as having a streamer in each live streaming room, and the number of live streaming rooms defined as having a streamer in each live streaming room is M, where N≧M≧2, and each of the M live streaming rooms collects the human body position information and human body appearance information of each streamer, and the streamer's voice information and voice position information, processes them into 3D avatar information, and sends them to the cloud service side; Step 6: The cloud service side imports the body position information and 3D avatar information of each streamer in each live streaming room and the voice information and voice position information of the streamer into the virtual scene in real time to form a VR data stream, and the cloud service side transmits the VR data stream to each live streaming device, and each streamer wears AR glasses, which are the display parts of the live streaming device, in the corresponding live streaming room. At this time, the virtual avatars of all streamers wearing AR glasses gather in the virtual scene, and the real avatars and virtual avatars of the streamers in each live streaming room overlap, and the streamers can only see the VR virtual avatars of other streamers through the AR glasses, and of course, when the cloud service side transmits the VR data stream to each live streaming device, the virtual avatars of the streamers can be omitted, and the streamers can only see the virtual avatars of other streamers through the AR glasses.

[0007] The generation of the 3D avatar information includes the steps of: S1): In each live streaming room, at least three video cameras are set up facing each streamer, and at least three video cameras synchronously track and shoot the streamers in the live streaming room, and synchronous tracking and shooting means that the time of each frame shot by different video cameras is the same; S2): The live streaming device performs chroma keying of the human body from different angles for each frame from videos shot by different video cameras in the same live broadcast room, and synthesizes a stereoscopic image of the human body. S3): The live streaming device transmits the stereoscopic image of the human body to the cloud service side, and the cloud service side recognizes the stereoscopic image of the human body frame by frame, records the video frames including the AR glasses with the stereoscopic image of the human body, and fuses the facial image and the stereoscopic image of the human body in the same time frame to form 3D avatar information.

[0008] The generation of the 3D avatar information includes the steps of: S1): In each live streaming room, at least three video cameras are installed facing each streamer, and the at least three video cameras synchronously track and shoot the streamer in the live streaming room, and synchronous tracking and shooting means that the time of each frame shot by different video cameras is the same; the streamer wears AR glasses in the live streaming room, and the AR glasses are equipped with a camera facing the face, and the time frame of the camera is synchronized with the time frame of the video camera, and the camera shoots the streamer's face, mainly shooting the part of the person's face that is blocked by the AR glasses; S2): The live streaming device sequentially fuses videos shot by multiple sets of cameras according to time frames and numbers to form a facial image, and the live streaming device also performs chroma keying of the human body from different angles for each frame from videos shot by different video cameras in the same live streaming room to synthesize a stereoscopic image of the human body without AR glasses; S3): The live streaming device transmits the face image and the stereoscopic image of the human body to the cloud service side, and the cloud service side recognizes the stereoscopic image of the human body frame by frame, records the video frame including the AR glasses in the stereoscopic image of the human body, and fuses the face image and the stereoscopic image of the human body of the same time frame to form 3D avatar information without the AR glasses.

[0009] In a live streaming room, a streamer wears AR glasses with cameras facing their face, the cameras are positioned around the lenses of the AR glasses, each camera shoots in sync, there are overlapping areas between the shooting ranges of adjacent cameras and also overlapping areas with video cameras, the cameras shoot in sync, and the live streaming device extracts facial information from the cameras for each frame.

[0010] The human body position information includes position coordinates and posture, the live streaming device collects position coordinates and postures of each streamer in each live streaming room and transmits the collected position coordinates and postures to a cloud service side, the live streaming device synchronously collects coordinates and postures of all people in the live streaming room in real time according to a time series of frames captured by a video camera, the position coordinates are collected through a positioning device, and posture data is collected through a gyroscope, and the positioning device and gyroscope are fixed to the chest of the streamer.

[0011] The audio information is collected through a recording device of a live streaming device, and the audio location information is collected through a sound source location device of the live streaming device.

[0012] The system includes a cloud service side and a live streaming device, wherein a cloud service processor is provided on the cloud service side and a terminal processor is provided on the live streaming device, and the live streaming device further includes at least three sets of video cameras, AR glasses, inner-ear wireless earphones, a positioning device, and a gyroscope, wherein the cloud service processor and the terminal processor are connected through TCP / IP communication, the video cameras, the VR head-mounted display, the inner-ear wireless earphones, the positioning device, and the gyroscope are electrically connected to the terminal processor, the positioning device is fixed and integrally installed with the gyroscope, and the positioning device and the gyroscope are fixed to the chest of the streamer.

[0013] The AR glasses include a spectacle frame and a VR display device provided in the spectacle frame. The VR display device includes a display screen and a convex lens inside the display screen. The convex lens and the display screen are combined to form a VR image to the human eye. The display screen is a transmissive display screen. A concave lens having a refractive index matching that of the convex lens is provided outside the transmissive display screen. The concave lens is used to offset the refraction of light by the convex lens and is located within the focal point of the convex lens. The convex lens is located within a virtual focal point of the concave lens. The light-transmitting portion of the transmissive display screen forms a realistic transparent image.

[0014] The transmissive display screen is a one-sided light-transmitting display screen that faces inward, i.e., toward the eyes, to display a VR image.

[0015] The one-sided light-transmitting display screen comprises an array of light-emitting areas, between which are light-transmitting areas of the array, the light-emitting areas emitting light from one side.

[0016] Each of the light-transmitting regions is a part of a Fresnel concave lens, and the array consisting of all the light-transmitting regions constitutes one Fresnel concave lens, which is installed on the outside of the transmission type display screen and replaces the concave lens whose refractive index matches that of the convex lens, making it possible to make the glass lighter and thinner.

[0017] The convex lens is a Fresnel convex lens, and the concave lens is a Fresnel concave lens.

[0018] The material of the light-transmitting region is a photosensitive color-changing light-transmitting material. When the luminescent material of the light-emitting region surrounding the light-transmitting region emits light, the color of the photosensitive color-changing light-transmitting material becomes darker and the light transmittance decreases. When the luminescent material of the light-transmitting region surrounding the light-transmitting region does not emit light, the light transmittance of the photosensitive color-changing light-transmitting region increases.

[0019] Between the convex lens and the display screen, a distance adjustment device is provided for adjusting the distance from the convex lens to the display screen.

[0020] The present invention discloses a method and system for performing realistic social activities using a virtual scene. By creating a unified coordinate system, the coordinate systems set in different live streaming rooms and the coordinate system created in the virtual scene are defined as three-dimensional coordinate systems with the same direction, providing a basis for realizing realistic social activities in the virtual scene, and then extracting the appearance information of streamers in different live streaming rooms and the corresponding three-dimensional coordinate position information of the streamers in the live streaming rooms, processing the extracted appearance information of the streamers into a three-dimensional avatar of a real person, and placing the three-dimensional avatar of the streamer in the virtual scene based on the three-dimensional coordinate position information of each person in the live streaming room, so that the coordinates of the streamer in the virtual scene and the three-dimensional coordinate position of the streamer in the live streaming room are the same, solving the problem that the change in the position of the streamer in reality does not match the virtual scene, and the position, direction and speed of each live streamer on the floor in the real live streaming room can be consistent with the position, direction and speed of the live streamer on the floor in the virtual scene, and the movement on the floor in the virtual scene can be consistent with the position, direction and speed of the live streamer on the floor in the virtual scene. If the movement is the same as on the floor, and the virtual objects in the virtual scene are bypassed, the real objects in the live streaming room are not changed, so of course the stairs on the ground in the virtual scene must also be real in the live streaming room to prevent people from stepping through them. For example, when live streamer A is live streaming in the live streaming room, he sees the real live streaming virtual avatars of other live streamers in the virtual scene through VR equipment, and wants to communicate with live streamer B, he can greet live streamer B in the virtual scene, and this greeting process is live streamed in the virtual scene, live streamer B finds live streamer A who is greeting in the virtual scene and responds to live streamer A, and live streamer A and live streamer B approach each other and commune with the virtual avatar of the other party's real live streaming. As long as the communication between the two is non-contact, such as conversation, gestures, facial expressions, etc., it can be realized, which is helpful to accurately establish the connection between the streamer and other people in the virtual scene and improve social efficiency. Effect of the Invention

[0021] In addition, since the present invention displays live-streamed real avatars in the virtual scene, it has better immersion and interaction, and the experience effect is also better. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram of a VR head-mounted display. [Diagram 2] FIG. 1 is a schematic diagram of the position of a limiting plate in a VR head mounted display. [Diagram 3] This is a schematic diagram of the camera arrangement inside the VR head mounted display. [Figure 4] FIG. 1 is a schematic diagram of a full-face VR head-mounted display. [Diagram 5] This is a schematic diagram of the camera arrangement inside a full-face VR head-mounted display. [Figure 6] FIG. 1 is a schematic diagram of a system for realizing realistic social activities in a virtual scene. [Figure 7] FIG. 1 is a diagram illustrating the principle of the AR glasses of the present invention. [Figure 8] 1 is a schematic diagram illustrating the configuration of AR glasses according to an embodiment. [Figure 9] FIG. 1 is a partial cross-sectional view of an AR glass lens according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The AR glasses are equipped with a spectacle frame 105 and a VR display device provided in the spectacle frame 105 as shown in Fig. 8, and the principle is as shown in Fig. 7. A display screen 121 and a convex lens 122 inside the display screen 121 constitute a VR display device, and a VR image is formed in the human eye. The display screen 121 is a transmissive display screen 1211, and a concave lens 124 whose refractive index matches that of the convex lens 122 is provided outside the transmissive display screen 1211. The concave lens 124 is used to offset the refraction of light by the convex lens and is located within the focal point of the convex lens 121, and the convex lens 121 is located within the virtual focal point of the concave lens 124, and the light-transmitting part of the transmissive display screen forms a real transparent image.

[0024] The transmissive display screen is a one-sided light-transmitting display screen that faces inward, i.e., toward the eyes, to display a VR image.

[0025] 8 and 9, the one-sided light-transmitting display screen has an array of light-emitting areas 102, and between the array of light-emitting areas 102 are array of light-transmitting areas 103, and the array of light-emitting areas 102 are fixed on a transparent bottom plate 104, and the parts of the transparent bottom plate 104 corresponding to the light-transmitting areas 103 are also light-transmitting areas, so the light-transmitting areas 103 can be holes, or can be filled with transparent material. The light-emitting areas emit light from one side.

[0026] Each light-transmitting region 103 is not a hole but a part of a Fresnel concave lens, and the array of all the light-transmitting regions forms a Fresnel concave lens of reduced transparency that is placed on the outside of the transmissive display screen and replaces a concave lens 106 whose refractive index matches that of the convex lens 101. Because the light-emitting regions 102 are blocked, the array of light-transmitting regions forms a Fresnel concave lens that transmits only half of the light, reducing its transparency by half.

[0027] Each light-transmitting area 103 is a hole, and the transparent bottom plate 104 is a complete Fresnel concave lens. If the light-emitting area 102 is not light-transmitting, it will block the light from the corresponding Fresnel concave lens part, and the corresponding transparent bottom plate 104 of each hole light-transmitting area 103 is a part of a Fresnel concave lens, and the light-transmitting part of the transparent bottom plate 104 forms a Fresnel concave lens with reduced transparency, which is provided on the outside of the transmission type display screen and replaces the concave lens 106 whose refractive index matches that of the convex lens 101. At this time, the convex lens 101 can be a Fresnel convex lens, and in an embodiment where the transparent bottom plate 104 is not a Fresnel concave lens, the concave lens is a Fresnel concave lens. These embodiments allow the glass to be made lighter and thinner.

[0028] The material of the light-transmitting region is a photosensitive color-changing light-transmitting material. When the luminescent material of the light-emitting region 102 around the light-transmitting region emits light, the color of the photosensitive color-changing light-transmitting material becomes darker and the light transmittance decreases, making the actual VR image in the light-emitting region 102 clearer. When the luminescent material around the light-transmitting region does not emit light, the light transmittance of the photosensitive color-changing light-transmitting region is high, making the real scene clearer through the light-transmitting region, and the combination of virtual and reality is more perfect.

[0029] A screw part 107 is provided between the convex lens 101 and the eyeglass frame 105, and the one-sided light-transmitting display screen is fixed on the eyeglass frame 105, so that the distance between the convex lens 101 and the eyeglass frame 105 can be adjusted to adjust the distance from the convex lens to the display screen. When wearing, the distance from the concave lens to the display screen is first adjusted so that the VR image is clear, and then the distance from the concave lens to the display screen is adjusted so that the light in the light-transmitting area passes through the concave lens and the convex lens and the real scene can be displayed clearly.

[0030] A camera facing the human eye can be provided on the temple of the eyeglass frame 105 to synthesize a three-dimensional image of a complete face. However, if the eyeglass frame is a thin eyeglass frame made of high-strength titanium alloy, for example, a truss-structure eyeglass frame using high-strength titanium alloy filament, the portion that can be blocked by the high-strength titanium alloy filament is very small, and the convex lens 101 of the lens is 10 to 25 mm away from the human eye, and images are taken at different angles simultaneously, and the blocked portion of the lens is also very small, the truss-structure eyeglass frame and lens using high-strength titanium alloy filament can be easily cut out to form a complete face shape.

[0031] Hereinafter, the technical means in the embodiments of the present invention will be described in detail with reference to the accompanying drawings of the present invention, but it goes without saying that the described embodiments are only some of the embodiments of the present invention and do not include all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without any creative activity are all within the scope of protection of the present invention.

[0032] The method for using the virtual scene to perform real social activities includes extracting avatars of streamers in live streaming rooms in different locations who participate in the real social activities, processing the extracted avatars of the streamers into three-dimensional avatars of real people, i.e., 3D real avatars, and placing the processed three-dimensional avatars or 3D real avatars at corresponding positions in the virtual scene according to the real-time position coordinates of each person to generate a virtual scene that can be interacted in real time, thereby enabling people in different geographical locations to perform immersive real social activities; before performing the real social activities, people in different locations enter the same virtual scene within the same time period, and realize the real social activities in the virtual scene.

[0033] A method for utilizing a virtual scene to perform realistic social activities includes the following steps.

[0034] Step 1: The cloud service side obtains the number N of live streaming rooms participating in the social virtual scene based on the number of live streaming devices, where the live streaming room is a live streaming scene in reality, and N≧2. For the social method, at least two live streaming scenes are required for the socialization to be meaningful. A live streaming device and a live streaming area are provided in the live streaming scene, and the live streaming device can extract appearance information and location information of a streamer in the live streaming area. The streamer can communicate and interact with the avatars of streamers in other live streaming rooms displayed in the social virtual scene through a VR head-mounted display in the live streaming area.

[0035] Step 2: Create N+1 3D coordinate systems based on the number N of live streaming rooms, each live streaming room creates one 3D coordinate system, and the N live streaming rooms constitute 1 to N 3D coordinate systems, and the cloud service side creates the N+1th 3D coordinate system in the virtual scene.

[0036] Step 3: Set the N+1 three-dimensional coordinate systems, and for each of the three-dimensional coordinate systems, the x-axis direction is due east, the y-axis direction is due north, and the z-axis direction is upward from sea level, with the units of length for each x-axis, y-axis, and z-axis being the same. This setting is to avoid interfering with the live streamer's sense of direction. Of course, if the N+1 three-dimensional coordinate systems are the same coordinate system and only the origin positions are different, coordinate transformation is required.

[0037] The floor of the virtual scene is defined as the sea level, and the floor of the live streaming room is also assumed to be the sea level. The spatial range occupied by the virtual scene in the N+1th 3D coordinate system is K N+1 The corresponding spatial range of each live streaming room in the 3D coordinate system is K 1 -K N and K 1 -K N There are no obstacles in K 1 -K NThe appearance of a real person in the live broadcast may be broadcast live in 3D, and the term "obstacle" refers to the degree of obstruction that prevents the completion of the live broadcast. Obstacles that can be filtered and corrected during the live broadcast are considered to be non-obstacles. With the above settings, the three-dimensional coordinate system corresponding to the live streaming room participating in the social virtual scene and the three-dimensional coordinate system created in the virtual scene have a unified coordinate system definition, that is, the directions of the x-axis, y-axis and z-axis of the three-dimensional coordinate system are the same, and the position coordinates of the people in the live streaming room can be mapped to the virtual scene in real time, so that the three-dimensional real avatars in the virtual scene have the same position coordinates as the people in the live streaming room.

[0038] Step 4: The cloud service side defines the origin position information of the N+1th 3D coordinate system in the virtual scene. The cloud server transmits the origin position information of the N+1th 3D coordinate system in the virtual scene to different live streaming devices, and determines the origin information of the 3D coordinate systems of the 1st to Nth live streaming rooms based on the origin position information received by the live streaming devices.

[0039] Step 5: There is at least one live streaming room defined as having a streamer in each live streaming room, and the number of live streaming rooms defined as having a streamer in each live streaming room is M, where N≧M≧2. In order for live streaming socializing to be meaningful, the number of live streaming rooms with a streamer in each live streaming room must be at least two. If M=1, there is only one live streaming room, there is no target for socializing, and the meaning of socializing is lost. The live streaming devices in the M live streaming rooms collect the location information and appearance information of each streamer and send it to the cloud service side, and the cloud service side processes the appearance information of each streamer in each live streaming room into 3D real avatar information.

[0040] Step 6: The cloud service side imports the 3D avatar information of each streamer in each live streaming room and the position information of each streamer in the live streaming room into the virtual scene in real time to form a VR data stream, and the cloud service side transmits the VR data stream to each live streaming device.

[0041] In step 5, in the live streaming room where people are present, the live streaming device collects the location information and appearance information of each streamer, and sends the appearance information of each streamer to the cloud service side, which processes the person's appearance information into realistic 3D avatar information, and imports the 3D avatar information into the virtual scene based on the location information corresponding to the 3D avatar information.

[0042] When there are two or more streamers in the live streaming room where the person is present, the live streaming device collects position information and appearance information of each streamer, processes the appearance information of each streamer into realistic 3D avatar information, imports the 3D avatar information into a virtual scene based on the position information corresponding to the 3D avatar, and collects two or more streamers separately to prevent two people from appearing in the same video and affecting subsequent extraction and separation.

[0043] When there is only one streamer in a live streaming room where the person is present, the location information and appearance information of the streamer are collected in each live streaming room respectively, the appearance information of each person is processed into realistic 3D avatar information, and the 3D avatar information is imported into the virtual scene based on the location information corresponding to the 3D avatar.

[0044] In step 5 of the method for performing realistic social activities using the virtual scene, in a live streaming room where people are present, the live streaming device collects audio information and audio position information of each streamer, and sends the audio information and audio position information of each streamer to the cloud service side, which processes the people's audio information and audio position information into realistic 3D audio information and audio position information, and imports the 3D audio information and audio position information into the virtual scene based on the position information corresponding to the 3D audio information and audio position information.

[0045] The location information includes location coordinates and posture, and the live streaming device collects the location coordinates and posture of each person in each live streaming room and transmits the collected location coordinates and posture to the cloud service side. The live streaming device synchronously collects the coordinates and posture of each person in the live streaming room in real time based on the time series of frames captured by a video camera.

[0046] The generation of realistic 3D avatar information includes the following steps: Step S1): In each live streaming room, at least three video cameras are installed facing each streamer, and the at least three video cameras synchronously track and shoot the streamers in the live streaming room, and synchronous tracking and shooting means that the time of each frame shot by different video cameras is the same. The video cameras are preferably RGBD video cameras, and the live streaming room is preferably a live streaming room with a green screen background, which is useful for subsequent chromakeying, Step S2): The streamer wears a VR head-mounted display equipped with multiple sets of cameras in a live streaming room, the multiple sets of cameras are numbered in sequence, the time frames of the multiple sets of cameras are synchronized with the time frames of the video cameras, and the multiple sets of cameras capture the streamer's face, and the capture includes a capture to determine the sound source position; Step S3): The live streaming device sequentially fuses the videos shot by the multiple sets of cameras according to time frames and numbers to form a face image, and the live streaming device also performs chromakeying of the human body from different angles for each frame from the videos shot by different video cameras in the same live streaming room to synthesize a stereoscopic image of the human body; Step S4): The live streaming device transmits the face image and the stereoscopic image of the human body to the cloud service side, and the cloud service side recognizes the stereoscopic image of the human body frame by frame, records the video frames including the VR head-mounted display in the stereoscopic image of the human body, and fuses the face image and the stereoscopic image of the human body of the same time frame to form 3D avatar information without the VR head-mounted display.

[0047] The captured video also includes audio information of the streamer in the live streaming room, the audio information includes audio content, the location of the sound source and the direction of the sound, and the cloud server extracts the audio content, the location of the sound source and the direction of the sound from the captured video, and transmits the audio to different live streamers in the virtual scene according to the location of the sound source and the direction of the sound, so that the live streamers in each live streaming room can hear the audio information of different positions, directions and intensities. When necessary, a gyroscope can be hidden in the hair of the live streamer to confirm the direction of the sound, eliminating the need for software and hardware to confirm the direction of the sound, and it is more economical to confirm the height of the sound source based on the height of the live streamer.

[0048] The VR head-mounted display includes a general VR head-mounted display, a full-face VR head-mounted display, and a projection-type virtual reality VR glass.

[0049] The projected virtual reality VR glasses are a conventional technology, and Patent Document 2 discloses projected virtual reality VR glasses.

[0050] The streamer wears projection-type virtual reality (VR) glasses in the scene, with at least two cameras mounted around each lens on the frame of the VR glasses, with an overlapping area between the shooting ranges of adjacent cameras, the cameras and a video camera shooting in sync, and the live streaming device extracts face information from the cameras for each frame.

[0051] Since it is a projection type display, the only part that is blocked is the eye part, so the blocked part is small, with a minimum value of only 2 to 5 square centimeters, and since there is a distance between the eye and the frame, the eye is not completely blocked, and color images of the eye can be fully captured. The live streamer wearing such VR glasses can see the floor of the live streaming room, which can eliminate the anxiety of stepping through, and makes it easier for people to accept such a social method. It can also feed back whether the floor of the live streaming room and the floor of the virtual scene overlap. If there is no overlap, information about the error range between the floor of the live streaming room and the floor of the virtual scene will be fed back and the information can be sent to the cloud service side, and the cloud service side can correct the error between the floor of the live streaming room and the floor of the virtual scene. In addition, a marker can be set on the floor of each live streaming room at the same coordinates, and a marker, for example a light-emitting point, can be set on the floor of the virtual scene at the same coordinates to monitor the overlapping status of each live streaming room and the marker in the virtual scene in real time, and the information can be sent to the cloud service side, and the cloud service side can correct the error between the floor of the live streaming room and the floor of the virtual scene.

[0052] As shown in FIG. 1 to FIG. 3, what the streamer wears in the live streaming room is a VR head mounted display, which includes a VR display screen 1, a lens fixing plate 2, a bracket 3, a holder 4, and a rubber band for mounting. The VR display screen 1 is symmetrically provided on one side of the holder 4, the lens fixing plate 2 is provided on the holder 4 on the opposite side of the VR display screen 1, a VR lens 5 is provided on the fixing plate 2, and brackets 3 are fixed on both side edges 4 of the lens fixing plate 2, which are arranged symmetrically. Adjacent to the bracket 3, a limiting plate 6 is provided on the side edge of the lens fixing plate 2 to limit the distance from the human eyes to the VR lens 5, so that there is a certain visual distance between the human eyes and the VR lens 5. A rubber band for mounting is also provided on the bracket 3, and the rubber band for mounting cooperates with the bracket 3 to allow the VR head mounted display to be smoothly mounted on a person's head. A camera is provided in the VR head mounted display, the camera is arranged on the fixed plate 2, the camera includes an upper camera group 7, a middle camera group 8, and a lower camera group 9, the upper camera group 7, the middle camera group 8, and the lower camera group 9 are evenly arranged, each camera shoots synchronously, there is an overlapping area between the shooting ranges of each two adjacent cameras, the camera and a video camera shoot synchronously, and the live streaming device extracts face information from the camera for each frame. The upper camera group 7, the middle camera group 8, and the lower camera group 9 can capture all the eye expressions of the streamer in the VR head mounted display.

[0053] The expression of the inner eye area of ​​the streamer is stitched together from videos taken by each camera in the upper camera group 7, the middle camera group 8, and the lower camera group 9. Since there is a certain overlapping area in the shooting range between adjacent cameras, the cameras on the fixed plate 2 are numbered in a certain order, for example, numbering each camera from left to right and top to bottom, or numbering each camera from bottom to top and right to left, and the left-right, top-bottom order is the left-right, top-bottom order in FIG.

[0054] Each frame image from the same time period of each camera is separated in order from the video according to the camera number order, and then, according to the camera number order, images with the same overlapping area are stitched together according to their position within the image by referring to the overlapping area with the overlapping area as a reference position, and finally, the stitching of the entire image is completed. The images after stitching of each frame is completed are then synthesized into a video, thereby obtaining eye expression information in the VR head mounted display.

[0055] In this embodiment, the numbering order is preferably from top to bottom and left to right, and the first camera 10 on the left side of the upper camera group 7 in Fig. 3 is numbered as A, the second camera 11 on the left side is numbered as B, and the remaining cameras are numbered in English alphabetical order according to their position. During stitching, in the same time frame, first extract the first image information PA taken by the camera with number A, then extract the second image information PB taken by the camera with number B, compare the positions of the overlapping areas of the first image information PA and the second image information PB in the image, and then cover the overlapping area of ​​the second image information PB with the overlapping area of ​​the first image information PA to complete the stitching of the two images, and sequentially stitch the image information taken by the remaining cameras, thereby stitching the images from all the cameras in the same time frame to form the streamer's eye expression image.

[0056] The camera in the VR head mounted display and the video camera in the live streaming room are also synchronously photographed, and the VR head mounted display is filtered from the appearance information of the streamer photographed by the video camera in the live streaming room based on the appearance characteristics of the VR head mounted display, and the appearance information of the streamer after filtering the VR head mounted display with reference to the actual face information of the streamer when not wearing the VR head mounted display is merged with the human eye expression image in the same time frame to form a real-time 3D avatar of the streamer when not wearing the VR head mounted display. The filtering method of the VR head mounted display performs grayscale conversion on each frame image in the appearance information of the streamer photographed by the video camera, identifies an area including the VR head mounted display in the grayscale image, determines a fixed point coordinate on each frame grayscale image, and uses the Matlab function imcrop to crop the image to complete the filtering of the VR head mounted display.

[0057] As shown in FIG. 4-FIG. 5, what a person wears in the scene is a full-face VR head-mounted display, which includes a VR display screen 1, a holder 4, and a face shield mask 12. The VR display screen 1 is provided on one side of the holder 4, and the face shield mask 12 is provided on the holder 4 on the opposite side of the VR display screen 1. The face shield mask 12 can cover the entire face of a person, and a VR lens is provided in the face shield mask 12. A camera and a light source are also provided in the face shield mask 12. The illuminance standard of the light source is 10lx-30lx, and the illuminance value of 10lx-30lx is a low light amount that does not stimulate the human eyes of the streamer. The provided light source helps the camera to capture the person's face. The cameras are evenly distributed in a number of rows and columns in the face shield mask 12, and preferably five rows and three columns in this embodiment. The cameras in the face shield mask shoot in a synchronized manner to extract the facial expressions of the person in the face shield mask, and there is an overlapping area in the shooting range between adjacent cameras in the face shield mask 12, and the cameras and video camera shoot in a synchronized manner, and the live streaming device extracts facial information from the cameras for each frame.

[0058] Since there is a certain overlapping area in the shooting ranges of adjacent cameras, the cameras on the fixed plate 2 are numbered in a certain order, for example, from left to right and top to bottom, or from bottom to top and right to left, and the left-right, top-bottom order is the left-right, top-bottom order in Figure 5.

[0059] Each frame image from the same time period of each camera is separated in sequence from the video according to the camera number order, and then, according to the camera number order, the overlapping area is used as the reference position, and images with the same overlapping area are stitched according to their position within the image by referring to the overlapping area, and finally, the stitching of the entire image is completed, and the images with each frame stitched are synthesized into a video, thereby obtaining facial expression information of the streamer inside the face shield mask.

[0060] The camera in the face shield mask 12 and the video camera in the live streaming room are also synchronized to take pictures, and a filtered full-face VR head-mounted display is filtered from the appearance information of the streamer taken by the video camera in the live streaming room based on the appearance characteristics of the VR head-mounted display, and the appearance information of the streamer after filtering the filtered full-face VR head-mounted display is merged with the facial expression image of the streamer in the same time frame by referring to the actual facial information of the streamer when not wearing the filtered full-face VR head-mounted display, to form a real-time 3D avatar of the streamer when not wearing the filtered full-face VR head-mounted display.

[0061] Although the full-face VR head-mounted display can completely cover the area of ​​a person's face, the extraction of the streamer's facial expressions is not as rich and accurate as that of a full-face VR head-mounted display because the VR head-mounted display only covers the eye area.

[0062] A system for performing real social activities using a virtual scene, the system includes a cloud service side 15 and a live streaming device 17, the cloud service side 15 includes a cloud server cluster 16, the live streaming device 17 includes a terminal processor 19, the live streaming device 17 further includes at least three sets of video cameras 18, a VR head mounted display 20, an inner-ear type wireless earphones 23, a positioning device 22, and a gyroscope 21, the cloud server cluster 16 and the terminal processor 19 are connected through TCP / IP communication, the video camera 18, the VR head mounted display 20, the inner-ear type wireless earphones 23, the positioning device 22, and the gyroscope 21 are electrically connected to the terminal processor 19. The positioning device 22 is fixed and integrally installed with the gyroscope 21, Velcro (registered trademark) is provided on the positioning device 22 and the gyroscope 21, and the positioning device 22 and the gyroscope 21 are fixed to the chest of a streamer via the Velcro (registered trademark).

[0063] The cloud service side 15 is used to generate a virtual scene and receive information transmitted from each live distribution room. The cloud service side 15 receives in real time the appearance information of the person transmitted from the video camera 18, the face information transmitted from the VR head mounted display 20, the coordinate information transmitted from the positioning device 22, and the posture information of the person transmitted from the gyroscope 21. The cloud service side 16 synthesizes a VR video of the real-time movement of the real person in the virtual scene, and transmits the VR video to the VR head mounted display 20.

[0064] It is preferable that the video camera 18 is an RGBD video camera, and multiple RGBD video cameras are fixed in each live streaming room, and the VR head-mounted display 20 is a general VR head-mounted display, a full-face VR head-mounted display, or a projection-type virtual reality VR glass.

[0065] The positioning device 22 includes a live streaming room origin positioning device and a wearable positioning device, the origin positioning device is an RTK base station, the wearable positioning device is provided with an RTK positioning module and a microcontroller, the RTK base station includes an RTK positioning module, an RTK-GPS antenna, and a data transceiver module, and transmits the observation value and the coordinate of the measurement point to the wearable positioning device through the data transceiver module and the RTK-GPS antenna, and the RTK positioning module in the wearable positioning device receives the observation value and the coordinate of the measurement point, collects GPS observation data, forms a differential observation value and processes it in real time, gives a centimeter-level positioning result, and transmits it to a server cluster through a terminal processor. The RTK positioning method is disclosed in Patent Document 3.

[0066] The gyroscope 21 can obtain the posture of the human body in real time, and is connected to the terminal processor via wireless communication.

[0067] The live distribution method can also be performed using a conventional live distribution method.

[0068] In the system for performing real social activities using the virtual scene, the number M of live streaming rooms in which the people are present is 5, a wrist positioning device is attached to the right wrist or the left and right wrist of each streamer, and a wearable device (including a chest-mounted positioning device corresponding to the positioning device and a right wrist-mounted positioning device or a left and right wrist-mounted positioning device corresponding to the wrist positioning device for right wrist or left and right wrist) corresponding to the virtual person imitated by the streamer is provided, and the position coordinates of the chest-mounted positioning device and the right wrist-mounted positioning device or the left and right wrist-mounted positioning device are transmitted to the cloud service side in real time, and the cloud service processor on the cloud service side compares the three-dimensional information of the positioning device and the chest-mounted positioning device in real time and issues an instruction to the wearable device to correct the position information of the chest in real time. The position coordinates of the wrist positioning device attached to the right wrist or the left and right wrist of the streamer are transmitted to the cloud service side, and the cloud service processor on the cloud service side compares the three-dimensional information of both and issues an instruction to the wearable device to correct the position information of the wrist positioning device attached to the right wrist or the left and right wrist in real time. A mouth-mounted voice generating device is provided, and a voice corresponding to the streamer on the cloud service side is transmitted to the mouth-mounted voice generating device, and the voice of the streamer is generated from the mouth-mounted voice generating device. In this way, the imitator who imitates the streamer can imitate the streamer in the virtual scene by wearing a wearable device and wearing a VR head-mounted display. Since the imitator is not live-streamed, he or she does not appear in the virtual scene, and the imitator's virtual person is hidden. As long as the position of the imitator's hands overlaps with the position of the corresponding virtual hands of the virtual person to be imitated, the imitator can imitate the streamer and shake hands with the real person being live-streamed. The imitator has the same body shape as the streamer, and after training, the imitator's movements match those of the streamer, allowing the imitator to interact with the real person being live-streamed in the form of tapping on the shoulder or hugging them. In this case, the live stream needs to filter the imitator and have them come into contact with each other in the virtual scene, and then stitch the three-dimensional images of the streamer's bodies into three-dimensional images of the streamers coming into contact with each other.In this way, the scene where the streamers are in contact with each other is displayed in the virtual scene, making the virtual scene richer, and the virtual scene can also be converted into 2D images from different angles for live streaming, which is particularly suitable for situations such as long-distance meetings. Here, the reason why M is limited to 5 or less is because there is a concern that too many imitators will cause problems and make it impossible to complete 3D live streaming. Even if M is 5 or less, it will not affect 3D live streaming, but if M is greater than 5, occlusion can be removed by filtering and compensation, and 3D live streaming can be performed normally, so M is not limited to 5 or less.

[0069] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above embodiments, but also include technical means consisting of any combination of the above technical features. Those skilled in the art can add some improvements and embellishments without departing from the principle of the present invention, and such improvements and embellishments are also considered to be within the protection scope of the present invention.

Claims

1. A method for performing real social activities using a virtual scene comprising a cloud service side and a live streaming device, comprising: Step 1: The cloud service side obtains the number N of live streaming rooms participating in the social virtual scene according to the number of the live streaming devices, and the live streaming room is a live streaming scene in reality, where N≧2; Step 2: Create N+1 identical three-dimensional coordinate systems based on the number N of the live streaming rooms, and each of the live streaming rooms creates one three-dimensional coordinate system, and the N live streaming rooms constitute 1 to N three-dimensional coordinate systems, and the cloud service side creates the N+1th three-dimensional coordinate system in the virtual scene; Step 3: Set the N+1 three-dimensional coordinate systems, each of which is composed of an x-axis, a y-axis, and a z-axis, and each of the x-axis, y-axis, and z-axis has the same length unit; The floor of the virtual scene is defined as a plane formed by the x-axis and y-axis of each three-dimensional coordinate system, and the floor of the live streaming room is also a plane formed by the x-axis and y-axis of the three-dimensional coordinate system. The spatial range occupied by the virtual scene in the N+1th three-dimensional coordinate system is K Ｎ＋１ The corresponding spatial range of each live streaming room in the three-dimensional coordinate system is K １ -K Ｎ and said K １ -K Ｎ There are no obstacles in K １ -K Ｎ The appearance of a real person in the live broadcast may be broadcast live in 3D, and the term "obstacle" refers to the degree of obstruction that prevents the completion of the live broadcast. Obstacles that can be filtered and corrected during the live broadcast are considered to be non-obstacles. Step 4: There is at least one live streaming room in which a streamer is defined as having a person, and the number of live streaming rooms in which a person is defined as having a person is M, where N≧M≧2. Each of the M live streaming rooms collects the human body position information and human body appearance information of each streamer, and the voice information and voice position information of the streamer, and transmits them to the cloud service side, and the cloud service side processes the appearance information of each streamer in each live streaming room into the 3D avatar information; Step 5: Replace step 4, and each of the live streaming rooms has at least one live streaming room defined as having a person in it, and the number of the live streaming rooms defined as having a person in it is M, where N≧M≧2, and each of the M live streaming rooms collects the human body position information and the human body appearance information of each of the streamers, and the voice information and the voice position information of the streamers, processes them into the 3D avatar information, and transmits them to the cloud service side; Step 6: The cloud service side imports the human body position information and the 3D avatar information of each of the streamers in each of the live streaming rooms and the voice information and voice position information of the streamers into the virtual scene in real time to form a VR data stream, and the cloud service side transmits the VR data stream to each of the live streaming devices, and each of the streamers wears AR glasses, which are display components of the live streaming device, in the corresponding live streaming room. At this time, the virtual avatars of all the streamers wearing the AR glasses gather in the virtual scene, and the real avatars of the streamers in each of the live streaming rooms overlap with the virtual avatars, and the streamers can only see the VR virtual avatars of the other streamers through the AR glasses. Of course, the cloud service side can drop the virtual avatars of the streamers when transmitting the VR data stream to each of the live streaming devices, and the streamers can only see the virtual avatars of the other streamers through the AR glasses. This includes the step A method for utilizing a virtual scene to perform realistic social activities.

2. The generation of the 3D avatar information includes: Step S1): In each of the live streaming rooms, at least three video cameras are installed facing each of the streamers, and the at least three video cameras synchronously track and shoot the streamers in the live streaming rooms, and synchronous tracking and shooting means that the time of each frame shot by different video cameras is the same; Step S2): The live streaming device performs chroma keying of a human body from different angles for each frame from videos shot by different video cameras in the same live distribution room, and synthesizes a stereoscopic image of the human body; Step S3): The live streaming device transmits the stereoscopic image of the human body to the cloud service side, and the cloud service side recognizes the stereoscopic image of the human body frame by frame, records a video frame including the AR glasses in the stereoscopic image of the human body, and fuses a face image and the stereoscopic image of the human body in the same time frame to form the 3D avatar information. This includes the step A method for conducting realistic social activities utilizing the virtual scene of claim 1.

3. The generation of the 3D avatar information includes: Step S1): In each of the live streaming rooms, at least three video cameras are installed facing each of the streamers, and the at least three video cameras synchronously track and shoot the streamers in the live streaming rooms, and synchronous tracking and shooting means that the time of each frame shot by the different video cameras is the same; the streamer wears the AR glasses in the live streaming room, and the AR glasses are installed with a camera facing the face, and the time frame of the camera is synchronized with the time frame of the video camera, and the camera shoots the face of the streamer, mainly shooting the part of the person's face that is blocked by the AR glasses; Step S2): The live streaming device sequentially fuses the videos taken by multiple sets of cameras according to time frames and numbers to form the face image, and the live streaming device performs chroma keying of the human body from different angles for each frame from the videos taken by different video cameras in the same live distribution room, and synthesizes a stereoscopic image of the human body without the AR glasses; Step S3): The live streaming device transmits the face image and the stereoscopic image of the human body to the cloud service side, and the cloud service side recognizes the stereoscopic image of the human body frame by frame, records a video frame including the AR glasses in the stereoscopic image of the human body, and fuses the face image and the stereoscopic image of the human body in the same time frame to form the 3D avatar information without the AR glasses. This includes the step A method for conducting realistic social activities utilizing the virtual scene of claim 2.

4. In the live streaming room, the streamer wears the AR glasses with cameras facing the face, the cameras are positioned around the lenses of the AR glasses, the cameras shoot in synchronization, there is an overlapping area between the shooting ranges of the adjacent cameras and also an overlapping area with the video camera, the cameras shoot in synchronization, and the live streaming device extracts face information from the cameras for each frame. A method for conducting realistic social activities utilizing the virtual scene of claim 3.

5. The human body position information includes position coordinates and posture, the live streaming device collects the position coordinates and postures of each of the streamers in each of the live streaming rooms and transmits the collected position coordinates and postures to the cloud service side, the live streaming device synchronously collects the coordinates and postures of all the people in the live streaming room in real time according to a time series of the frames captured by the video camera, the position coordinates are collected through a positioning device, the posture data is collected through a gyroscope, and the positioning device and the gyroscope are fixed to the chest of the streamer. A method for conducting realistic social activities utilizing the virtual scene of claim 1.

6. The audio information is collected through a recording device of the live streaming device, and the audio location information is collected through a sound source location device of the live streaming device. A method for conducting realistic social activities utilizing the virtual scene of claim 1.

7. A system for performing real social activities using a virtual scene according to any one of claims 1 to 6, comprising: The present invention comprises a cloud service side and a live streaming device, the cloud service side being provided with a cloud service processor, the live streaming device being provided with a terminal processor, the live streaming device further comprising at least three sets of video cameras, AR glasses, inner-ear type wireless earphones, a positioning device, and a gyroscope, the cloud service processor and the terminal processor being connected through TCP / IP communication, the video cameras, the VR head mounted display, the inner-ear type wireless earphones, the positioning device, and the gyroscope being electrically connected to the terminal processor, the positioning device being fixed and integrally installed with the gyroscope, and the positioning device and the gyroscope being fixed to the chest of the streamer. A system characterized by:

8. The AR glasses include a spectacle frame and a VR display device provided in the spectacle frame, the VR display device includes a display screen and a convex lens inside the display screen, the convex lens and the display screen are combined to form a VR image to the human eye, the display screen is a transmissive display screen, and a concave lens whose refractive index matches the refractive index of the convex lens is provided outside the transmissive display screen, the concave lens is used to offset the refraction of light by the convex lens and is located within the focus of the convex lens, the convex lens is located within a virtual focus of the concave lens, and the light-transmitting portion of the transmissive display screen forms a realistic transparent image. AR glasses characterized by:

9. The transmissive display screen is a one-sided light-transmitting display screen that is directed inward, i.e., toward the eye, to display a VR image.

10. AR glasses according to claim 8.

10. The one-sided light-transmitting display screen comprises an array of light-emitting regions, between which are array light-transmitting regions, the light-emitting regions emitting light from one side.

10. AR glasses according to claim 9.

11. Each of the light-transmitting regions is a part of a Fresnel concave lens, and an array consisting of all the light-transmitting regions constitutes one Fresnel concave lens. The Fresnel concave lens is provided on the outside of the transmissive display screen, and replaces the concave lens whose refractive index matches that of the convex lens. In this case, the glass can be made lighter and thinner.

10. AR glasses according to claim 8.

12. The convex lens is a Fresnel convex lens, and the concave lens is a Fresnel concave lens.

10. AR glasses according to claim 8.

13. The material of the light-transmitting region is a photosensitive color-changing light-transmitting material. When the light-emitting material of the light-emitting region around the light-transmitting region emits light, the color of the photosensitive color-changing light-transmitting material becomes darker and the light transmittance decreases. When the light-emitting material around the light-transmitting region does not emit light, the light transmittance of the photosensitive color-changing light-transmitting region increases. AR glasses according to claim 10 or 11.

14. A distance adjustment device is provided between the convex lens and the display screen for adjusting the distance from the convex lens to the display screen. AR glasses according to claim 13.

Images