A method and device for positioning a virtual character in an interactive system

Abstract

The present application relates to the field of AR technology, and provides a method and device for positioning a virtual character in an interactive system, which extracts semantic information of a real object and its distance from a rendering terminal from an environmental image of a captured real scene, and sends the information to a capture terminal. The capture terminal adjusts the initial position of the virtual character in a capture coordinate system according to the received semantic information of the real object and its distance from the rendering terminal, and synchronizes the adjusted target position to the rendering coordinate system. Through a two-way feedback mechanism between the rendering terminal and the capture terminal, the rendering terminal can accurately place the virtual object in the real scene when displaying the virtual object, which reduces the sense of disharmony of the fusion of the virtual character and the real scene to a certain extent, and improves the authenticity of the fusion of virtual and real.

Description

Technical Field

[0001] The present application relates to the field of augmented reality (AR) technology, and in particular to a method and device for positioning a virtual character in an interactive system. Background Art

[0002] AR technology is a technology that cleverly integrates simulated virtual information with the real world. With the development of AR technology and computer vision and computer graphics technology, the application of virtual characters in three-dimensional communication scenarios to achieve "enhancement" of the real world and improve the interactive experience has gradually become a new way of socializing.

[0003] At present, in actual AR interaction scenarios, the positioning methods of virtual characters mainly include the following: 1) placing the virtual character in a fixed position in the three-dimensional space according to the design of the virtual space; 2) strictly setting the relationship between the virtual character and the surrounding real objects in advance, and placing the virtual character according to the set relationship; 3) detecting the plane in the virtual space and placing the virtual character on the detected plane.

[0004] In the interactive system, when the rendering terminal renders and displays the virtual character reconstructed by the acquisition terminal, the position of the virtual character can be located in real time according to any of the above methods. However, since the virtual character reconstructed by the acquisition terminal is disconnected from the rendering terminal, the virtual character at the current position may not be accurately integrated with the real scene displayed by the rendering terminal and the objects in the real scene, which reduces the authenticity of the virtual-real integration and affects the user's AR experience. Summary of the invention

[0005] The embodiments of the present application provide a method and device for positioning a virtual character in an interactive system, which are used to improve the accuracy of positioning the virtual character, thereby improving the authenticity of the fusion of virtual and real.

[0006] On the one hand, an embodiment of the present application provides a method for positioning a virtual character in an interactive system, comprising:

[0007] Collect environmental images of real scenes and establish rendering coordinate systems;

[0008] When it is determined according to the environment image that a real object exists in the real scene, extracting a set of attribute information of the real object;

[0009] Sending the attribute information set to a collection terminal in the interactive system, so that the collection terminal adjusts the initial position of the virtual character of the target object in the collection coordinate system according to the attribute information set, and synchronizes the adjusted target position to the rendering coordinate system;

[0010] The received three-dimensional reconstruction data of the virtual character is rendered at a target position in the rendering coordinate system, and the rendered virtual character is superimposed and displayed in the environment image.

[0011] On the other hand, an embodiment of the present application provides a method for positioning a virtual character, which is applied to a collection terminal in an interactive system, including:

[0012] Receiving an attribute information set of a real object sent by a rendering terminal in an interactive system, wherein the attribute information set is extracted by the rendering terminal after determining that a real object exists in the real scene according to a captured environment image of the real scene;

[0013] According to the attribute information set, adjusting the initial position of the virtual character of the target object in the acquisition coordinate system, and synchronizing the adjusted target position to the rendering coordinate system, wherein the rendering coordinate system is established by the rendering terminal according to the environment image;

[0014] The three-dimensional reconstruction data of the virtual character of the target object is sent, so that the rendering terminal performs rendering at the target position of the rendering coordinate system according to the three-dimensional reconstruction data, and overlays and displays the rendered virtual character in the environment image.

[0015] On the other hand, an embodiment of the present application provides a rendering terminal, which is applied to a remote three-dimensional interactive system, including a camera, a processor, a memory, a display screen, and a communication interface, wherein the camera, the memory, the display screen, the communication interface, and the processor are connected via a bus:

[0016] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0017] Collect environmental images of real scenes through cameras and establish rendering coordinate systems;

[0018] When it is determined according to the environment image that a real object exists in the real scene, extracting a set of attribute information of the real object;

[0019] Sending the attribute information set to a collection terminal in the interactive system through the communication interface, so that the collection terminal adjusts the initial position of the virtual character of the target object in the collection coordinate system according to the attribute information set, and synchronizes the adjusted target position to the rendering coordinate system;

[0020] The three-dimensional reconstruction data of the virtual character sent by the acquisition terminal is received through the communication interface, the three-dimensional reconstruction data is rendered at a target position in the rendering coordinate system, and the rendered virtual character is superimposed and displayed in the environment image through the display screen.

[0021] On the other hand, an embodiment of the present application provides a collection terminal, which is applied to a remote three-dimensional interactive system, including a camera and a host, wherein the host includes a processor, a memory, a display screen, and a communication interface, and the camera, the memory, the display screen, the communication interface, and the processor are connected via a bus:

[0022] The display screen is used to display the reconstructed virtual character;

[0023] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0024] Receiving, via the communication interface, a set of attribute information of a real object sent by a rendering terminal in the interactive system, wherein the set of attribute information is extracted by the rendering terminal after determining that a real object exists in the real scene according to a captured environment image of the real scene;

[0025] According to the attribute information set, adjusting the initial position of the virtual character of the target object in the acquisition coordinate system, and synchronizing the adjusted target position to the rendering coordinate system, wherein the rendering coordinate system is established by the rendering terminal according to the environment image;

[0026] The three-dimensional reconstruction data of the virtual character of the target object is sent through the communication interface, so that the rendering terminal renders at the target position of the rendering coordinate system according to the three-dimensional reconstruction data, and overlays and displays the rendered virtual character in the environmental image, wherein the virtual character is reconstructed based on the data collected by the camera.

[0027] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the method for positioning a virtual character in an interaction process provided by an embodiment of the present application.

[0028] In a method and device for positioning a virtual character in an interactive system provided by an embodiment of the present application, the accuracy of positioning the virtual character is improved through a two-way feedback mechanism between a collection terminal and a rendering terminal in the interactive system, thereby improving the authenticity of virtual-real fusion. Specifically, the rendering terminal establishes a rendering coordinate system based on the collected environmental image of the real scene, extracts the attribute information set of the real object in the real scene and sends it to the collection terminal. The collection terminal adjusts the initial position of the virtual character of the target object in the collection coordinate system according to the received attribute information set, and synchronizes the adjusted target position to the rendering coordinate system. In this way, after the rendering terminal receives the three-dimensional reconstruction data of the virtual character sent by the collection terminal, the virtual character is rendered at the target position of the rendering coordinate system and superimposed on the environmental image, thereby accurately merging into the real scene. The position of the virtual character reconstructed by the collection terminal is assisted by the environmental information in the real scene of the rendering terminal, and synchronized to the rendering terminal, so that when the rendering terminal displays the virtual character at the synchronized target position, the abnormal phenomenon such as occlusion or floating of the virtual character and the real object can be reduced, which reduces the sense of disharmony of the fusion of the virtual character and the real scene to a certain extent, and improves the authenticity of virtual-real fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 An interactive system architecture diagram provided for an embodiment of the present application;

[0031] Figure 2A A schematic diagram of a live broadcast scenario provided in an embodiment of the present application;

[0032] Figure 2B A schematic diagram of a remote conference scenario provided in an embodiment of the present application;

[0033] Figure 3 A flowchart of a method for rendering a virtual character in a terminal positioning interaction system when a real object exists provided in an embodiment of the present application;

[0034] Figure 4 A flow chart of a method for establishing a rendering coordinate system for a rendering terminal provided in an embodiment of the present application;

[0035] Figure 5A A flow chart of a method for detecting a plane in an environmental image provided by an embodiment of the present application;

[0036] Figure 5BA flow chart of another method for detecting a plane in an environment image provided by an embodiment of the present application;

[0037] Figure 6 A display effect diagram of a rendering terminal provided in an embodiment of the present application;

[0038] Figure 7 A flowchart of a method for rendering a virtual character in a terminal positioning interaction system when a real object does not exist provided in an embodiment of the present application;

[0039] Figure 8 A flow chart of a method for collecting virtual characters in a terminal positioning interactive system when a real object exists provided in an embodiment of the present application;

[0040] Fig. 9A A flow chart of a method for adjusting the initial position of a virtual character by a collection terminal provided in an embodiment of the present application;

[0041] Fig. 9B A flowchart of another method for adjusting the initial position of a virtual character by a collection terminal provided in an embodiment of the present application;

[0042] Fig.10 A flowchart of a method for collecting virtual characters in a terminal positioning interactive system when there is no real object provided in an embodiment of the present application;

[0043] Fig.11 A schematic diagram of an interaction scenario between a collection terminal and a rendering terminal provided in an embodiment of the present application;

[0044] Fig.12 A flow chart of the interaction method between the acquisition terminal and the rendering terminal provided in the embodiment of the present application;

[0045] Fig.13 A diagram of the structure of a rendering terminal provided in an embodiment of the present application;

[0046] Fig.14 This is a structural diagram of the collection terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] At present, in an interactive system that uses virtual characters for three-dimensional communication, an acquisition terminal reconstructs the virtual character based on the acquired data, and sends the three-dimensional reconstruction data of the virtual character to a rendering terminal through a transmission terminal. The rendering terminal overlays and displays the virtual character on a real scene.

[0048] From the perspective of technical architecture, the interactive system mainly includes three parts: 3D reconstruction, data transmission, and rendering display. Among them, 3D reconstruction refers to the reconstruction of virtual characters based on human body data collected by sensors, data transmission refers to the encoding and decoding transmission of the 3D reconstruction data of virtual characters, and rendering display refers to the rendering and display of virtual characters based on the received 3D reconstruction data. Among them, the collected human body data is related to the type of sensor, which includes optical scanners (such as visible structured light or laser scanners), RGBD cameras, and RGB cameras.

[0049] Among the 3D holographic communication and interaction products that have been launched on the market, some use single-frame 2D images to generate realistic or cartoon-like 3D digital humans, but due to the small amount of model data, the simulation effect is not good. In the direction of virtual characters based on real people, some use a large number of camera arrays for 3D data collection and use greater computing power for modeling to generate 3D digital humans with better simulation effects, but they require wider network bandwidth and higher system costs; others use RGBD cameras to capture 3D geometric data and texture data to reconstruct 3D digital humans.

[0050] At present, the technology of using virtual characters for three-dimensional communication is in the development stage, and the products on the market are relatively rudimentary. In the positioning of virtual characters and the placement of models during rendering and display, the virtual characters are mainly placed in a preset fixed position, and there is no direct interaction with the objects in the real scene, which makes the reconstructed virtual characters disconnected from the rendering terminal, causing the rendering terminal to display the virtual characters in an inappropriate position, thereby blocking the objects in the real scene or floating in the real scene, reducing the reality of virtual fusion.

[0051] At the same time, the current positioning of virtual characters and the placement of models will also bring the following problems to the interactive system:

[0052] In the 3D reconstruction part: Considering the positional limitations of the virtual character when it is rendered and displayed on the rendering terminal, the sensor for collecting data needs to be strictly set, resulting in limitations on collection and reconstruction. For example, if the sensor is a camera, the camera's collection height, collection angle, and collection character's position need to be strictly set.

[0053] In the rendering and display part: Due to the limitation of the pre-set character position, the position of the virtual character cannot be accurately located, so the virtual character cannot be placed adaptively according to the real scene, and it may overlap with objects in the real scene, increasing the interaction loss. Therefore, the position of the virtual character needs to be adjusted. On the one hand, if the rendering terminal has a gesture recognition function, it is necessary to manually adjust the spatial position of the virtual character model and place it, which reduces the user experience. On the other hand, if the rendering terminal does not have a gesture recognition function, the virtual character can only be placed through a fixed viewpoint and fixed position, which reduces the user experience.

[0054] In view of this, the embodiment of the present application provides a method and device for positioning a virtual character in an interactive system, which improves the accuracy of virtual character positioning through a two-way feedback mechanism between a collection terminal and a rendering terminal, thereby improving the authenticity of virtual-real fusion. Specifically, the rendering terminal extracts the semantic information of the real object and the distance between the real object and the rendering terminal based on the collected environmental image of the real scene, and sends it to the collection terminal through the transmission terminal. When reconstructing the virtual character, the collection terminal assists in positioning the virtual character and synchronizes it to the rendering terminal based on the semantic information extracted from the real scene and the prompt and guidance of the distance, thereby enhancing the interactivity of the virtual character with the objects in the real scene, so that when the rendering terminal displays the virtual character based on the three-dimensional reconstructed data sent by the collection terminal, it can effectively prevent the virtual character from being blocked by the real object, or the virtual character from floating in the real scene and other abnormal phenomena, and then when the virtual character is used for AR interaction and three-dimensional holographic communication, the virtual character is better integrated with the real scene, and the character interaction experience is enhanced.

[0055] See also Figure 1 , is an overall architecture diagram of the interactive system provided in an embodiment of the present application, which mainly includes an acquisition terminal, a transmission terminal and a rendering terminal.

[0056] The acquisition terminal is used to collect the geometric data and texture data of the target object, and perform geometric 3D reconstruction and texture 3D reconstruction based on the collected geometric data and texture data, generate a virtual character based on a real person, and send the 3D reconstruction data of the virtual character to the rendering terminal. During the reconstruction process, the environmental information of the real scene of the rendering terminal can be fed back to the acquisition personnel through AR glasses. The acquisition personnel set the interaction range of the target object according to the environmental information and the visual range of the camera (RGB camera or RGBD camera) to adjust the initial position of the virtual character of the target object in the acquisition coordinate system corresponding to the acquisition terminal, and synchronize the adjusted target position to the rendering terminal, so that the rendering terminal can accurately place the virtual character, improve the authenticity of the fusion of virtual and real, and thus improve the user's immersive experience.

[0057] The acquisition terminal can be an interactive device consisting of an RGB camera or an RGBD camera and a host or workstation with computing power. This application does not impose any restrictive requirements on the interactive device of the acquisition terminal. For example, it can be a laptop, desktop, smart phone, AR head-mounted display device, AR glasses and other interactive devices.

[0058] The transmission terminal is used to obtain the three-dimensional reconstruction data sent by the acquisition terminal, and distribute it to the rendering terminal after encoding and decoding; and obtain the environmental information of the real scene fed back by the rendering terminal, and distribute it to the acquisition terminal after encoding and decoding.

[0059] Among them, the transmission terminal can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0060] The rendering terminal is used to collect environmental images of the surrounding real scene based on the equipped RGB camera, extract environmental information from the environmental images and feed it back to the acquisition terminal, so that the rendering terminal adjusts the initial position of the virtual character of the target object in the acquisition coordinate system corresponding to the acquisition terminal, and synchronizes the adjusted target position to the rendering terminal. In this way, after the rendering terminal renders the virtual character according to the three-dimensional reconstruction data sent by the acquisition terminal, it can accurately place the virtual character at the target position of the rendering coordinate system, thereby improving the authenticity of the fusion of virtual and real, and thereby improving the user's immersive experience.

[0061] The rendering terminal can be an AR head-mounted display device with AR function, AR glasses and other three-dimensional interactive devices. In addition, during the display of the virtual character, it can also be connected to two-dimensional display devices such as smart TVs, smart phones, and tablets for display.

[0062] The environmental information in the above embodiments refers to attribute information such as semantic information of a real object, a distance between the real object and a rendering terminal, and a local image of the real object.

[0063] It should be noted that in the embodiments of the present application, the three terminals can be deployed independently according to the actual interaction scenario.

[0064] For example, see Figure 2A In the live broadcast scenario shown, the host end is provided with a collection terminal in the interactive system, and the audience end is provided with a rendering terminal in the interactive system. The collection terminal and the rendering terminal transmit data through the cloud server of the live broadcast platform.

[0065] exist Figure 2AIn the live broadcast scene shown, the acquisition terminal at the host end is configured with a camera and a host. The camera is used to collect the host's image data, and the host is used to reconstruct the host's virtual character based on the image data, and send the 3D reconstruction data of the virtual character to the rendering terminal. In addition, AR glasses are used to receive the environmental information of the real scene at the audience end fed back by the rendering terminal, adjust the initial position of the host's virtual character in the acquisition coordinate system, and synchronize the adjusted target position to the rendering terminal. The AR glasses of the rendering terminal receive the 3D reconstruction data of the virtual character and display the virtual character at the target position to enhance the real scene. At the same time, AR glasses can be connected to display devices such as mobile phones, TVs, and tablets for 2D display.

[0066] For another example, see Figure 2B The conference scene shown in the figure, in each conference room where the remote conference is held (such as Figure 2B The acquisition terminal and rendering terminal in the interactive system are deployed in conference rooms 1-4) respectively.

[0067] exist Figure 2B In the conference scene shown, the functions of the acquisition terminal and the rendering terminal in each conference room are the same. Taking the acquisition terminal and the rendering terminal in conference room 1 as an example, the camera configured in the acquisition terminal is used to collect image data of user 1 in conference room 1. The host reconstructs the virtual character 1 of user 1 based on the image data collected by the camera, and sends the three-dimensional reconstruction data of the virtual character 1 to the rendering terminal in each conference room through the cloud server; the rendering terminal sends the environmental information of user 1 in the real scene to the acquisition terminal in each conference room through the cloud server, so that the acquisition terminal adjusts the initial position of each virtual character in the acquisition coordinate system and synchronizes it to the rendering terminal. At the same time, the AR glasses of the rendering terminal receive the three-dimensional reconstruction data of each virtual character, and place the rendered virtual characters according to their respective positions, so as to enhance the real scene, and can be connected to a TV, mobile phone, tablet, etc. for two-dimensional display.

[0068] It should be noted that Figure 2A and Figure 2B This is only a scenario example and does not limit the number of people interacting. Regardless of whether the acquisition terminal and the rendering terminal are deployed separately or together, the corresponding number of users can be one or more.

[0069] Based on the interactive system of the embodiment of the present application, Figure 3 The flowchart of the method for positioning a virtual character in an interactive system provided by an embodiment of the present application is exemplarily shown. The flow is applied to a rendering terminal in the interactive system and mainly includes the following steps:

[0070] S301: Collecting an environmental image of a real scene and establishing a rendering coordinate system.

[0071] The rendering terminal is equipped with cameras, including but not limited to binocular cameras and RGBD cameras, which can collect environmental images and depth data of real scenes.

[0072] When executing S301, after the rendering terminal starts the AR application, the head is rotated to drive the camera to scan the real scene around the rendering terminal, obtain the environment image of the real scene, and establish a rendering coordinate system based on the collected environment image. The establishment process of the rendering coordinate system is shown in Figure 4 , mainly includes the following steps:

[0073] S3011: Extract the three-dimensional point cloud data contained in the environment image.

[0074] In an optional implementation, an intrinsic shape signature (ISS) algorithm may be used to extract 3D point cloud data from an environmental image. The technology for extracting 3D point cloud data from an image is already quite mature, and this part is not the focus of this application and will not be described in detail.

[0075] In an optional implementation, when the camera installed on the rendering terminal is an RGBD camera, three-dimensional point cloud data can also be obtained based on depth data corresponding to the environment image.

[0076] S3012: Detect whether the environment image contains at least one plane based on the three-dimensional point cloud data. If so, execute S3013; otherwise, execute S3014.

[0077] S3013: Establish a rendering coordinate system using the detected target plane as the ground.

[0078] S3014: Re-collect the environment image and return to S3011.

[0079] At present, the commonly used algorithms for detecting planes from three-dimensional point cloud data mainly include: Random Sample Consensus (RANSAC) algorithm, Region Growing algorithm and Grid-based Region Growing algorithm.

[0080] Among them, the RANSAC algorithm, as the most classic algorithm, can calculate the mathematical model parameters based on a set of sample data sets containing abnormal data, thereby obtaining valid sample data. The basic assumption of the RANSAC algorithm is that the sample data set contains correct data (data that can be described by the mathematical model, also called inliers), and also contains abnormal data (data that deviates far from the normal range and cannot adapt to the data model, also called outliers), that is, the sample data set contains noise. These abnormal data may be caused by incorrect measurements, incorrect assumptions, incorrect calculations, etc. At the same time, the RANSAC algorithm also assumes that, given a set of correct data, there is a method to calculate the model parameters that conform to these data.

[0081] In the process of fitting a plane from 3D point cloud data, small concave and convex points on the plane (such as small potholes on the ground) can be regarded as valid data, which has a certain offset to the fitted plane, while concave and convex points with too large offset (such as obstacles on the ground, deep pits on the ground) can be regarded as invalid data. Therefore, the RANSAC algorithm can be used to detect whether the environment image contains at least one plane based on the 3D point cloud data. For the specific detection process, see Figure 5A , mainly includes the following steps:

[0082] S3012_11: Randomly fit multiple planes based on 3D point cloud data.

[0083] S3012_12: For each plane, determine the number of points in the three-dimensional point cloud data whose distance from the plane exceeds a preset distance threshold.

[0084] If the distance between the midpoint of the 3D point cloud data and the plane exceeds the preset distance threshold, it indicates that the point is invalid data. If the distance between the midpoint of the 3D point cloud data and the plane is less than the preset distance threshold, it indicates that the point is valid data.

[0085] S3012_13: Determine whether the number of points exceeding the preset distance threshold is greater than the set first quantity threshold, if so, execute S3012_14, otherwise, execute S3012_15.

[0086] If the number of points exceeding the preset distance threshold is greater than the set number threshold, it indicates that the plane is invalid and should be discarded. If the number of points exceeding the preset distance threshold is less than or equal to the set number threshold, it indicates that the plane is valid and should be retained.

[0087] S3012_14: Discard this plane.

[0088] S3012_15: Keep the plane.

[0089] S3012_16: From the retained planes, select a plane with the least number of points exceeding a preset distance threshold, and use the selected plane as the final detected target plane.

[0090] In S3012_16, there may be multiple planes that meet the above requirements. Among the retained planes, the plane with the least number of points exceeding the preset distance threshold indicates the best fitting effect and can be used as the plane included in the environment image.

[0091] In the embodiment of the present application, in addition to using invalid data in the three-dimensional point cloud data to detect the plane, the valid data can also be used to detect the plane. For the specific process, see Figure 5B , mainly includes the following steps:

[0092] S3012_21: Randomly fit multiple planes based on 3D point cloud data.

[0093] S3012_22: For each plane, determine the number of points contained in the plane whose distance is less than a preset distance threshold.

[0094] S3012_23: Determine whether the number of points less than the preset distance threshold is greater than the set second number threshold, if so, execute S3012_24, otherwise, execute S3012_25.

[0095] S3012_24: Keep the plane.

[0096] S3012_25: Discard this plane.

[0097] S3012_26: From the retained planes, select a plane with the least number of points less than a preset distance threshold, and use the selected plane as the final detected target plane.

[0098] Compared with least squares fitting, the RANSAC algorithm aims to obtain fitting results based on most of the valid data, so the plane detection effect is better.

[0099] After the rendering coordinate system is established, the collected environment image is used to detect whether the real scene where the rendering terminal is located contains real objects, so as to prevent the virtual character and the real object from being blocked during superimposed display. For the specific process, see S302 to S304.

[0100] S302: When it is determined that a real object exists in the real scene according to the environment image, an attribute information set of the real object is extracted.

[0101] In S302, the detection of real objects can adopt common target detection algorithms, such as convolutional neural network detection algorithms, deep neural network detection algorithms, etc., and the present application embodiment does not impose restrictive requirements. When the real object is detected, a partial image of the real object can be intercepted from the environment image through the detection frame.

[0102] After detecting the real object, the environment image is subjected to image semantic segmentation to extract the semantic information of the real object (such as the category of the real object). The present application embodiment does not impose restrictive requirements on the image-based semantic segmentation algorithm, including but not limited to the fully connected network (FCN) based on deep learning, U-net, SegNet, Deeplab, PSPNet, etc.

[0103] In the embodiment of the present application, since the camera configured for the rendering terminal is a binocular camera or an RGBD camera, the distance between the real object in the real scene and the rendering terminal can be analyzed.

[0104] According to the semantic information of the real object and the distance between the real object and the rendering terminal, the attribute information set of the real object is obtained, or according to the local image of the real object and the distance between the real object and the rendering terminal, the attribute information set of the real object is obtained.

[0105] S303: Sending the attribute information set to the acquisition terminal in the interactive system, so that the acquisition terminal adjusts the initial position of the virtual character of the target object in the acquisition coordinate system according to the attribute information set, and synchronizes the adjusted target position to the rendering coordinate system.

[0106] Generally, when reconstructing a virtual character, the acquisition terminal performs three-dimensional reconstruction based on the acquisition coordinate system of the acquisition terminal itself. That is to say, the reconstructed virtual character has an initial position in the acquisition coordinate system, which is not affected by the real scene in the rendering terminal.

[0107] In an embodiment of the present application, the rendering terminal sends a set of attribute information extracted from the real scene to the acquisition terminal. Since the attribute information set reflects the real information of the real objects in the real scene where the rendering terminal is located, the real scene on the rendering terminal side can be perceived according to the attribute information through the AR glasses on the acquisition terminal side. Therefore, for the reconstructed virtual character, the original position of the virtual character in the acquisition coordinate system can be adjusted according to the real scene on the rendering terminal side, and the adjusted position can be synchronized to the rendering coordinate system, thereby ensuring that when the rendering terminal displays the virtual character, there will be no occlusion with the real objects in the real scene, thereby improving the authenticity of the fusion of virtual and real.

[0108] Among them, the process of adjusting the position of the virtual character refers to the method flow of locating the virtual character on the acquisition terminal side, which will not be described in detail here.

[0109] S304: Rendering the received three-dimensional reconstruction data of the virtual character at a target position in a rendering coordinate system, and superimposing and displaying the rendered virtual character in an environment image.

[0110] In S304, after the acquisition terminal has reconstructed the virtual character, it sends the 3D reconstruction data of the virtual character to the rendering terminal through the transmission terminal, so that the rendering terminal renders the virtual character according to the 3D reconstruction data, and according to the design of the AR application, the rendered virtual character is superimposed and displayed in the environmental image of the real scene. During the display process, the target position of the virtual character in the rendering coordinate system is adjusted by the acquisition terminal and synchronized to the rendering terminal. Since the acquisition terminal refers to the information of the real objects in the real scene fed back by the rendering terminal when adjusting the initial position of the virtual character, it can ensure that the virtual character placed at the target position will not be blocked by the real objects in the real scene, thereby improving the authenticity of the virtual-real fusion.

[0111] In some embodiments, when the rendering terminal superimposes a virtual character on the environmental image through AR technology and displays it in the field of view, the movable range of the virtual character in the rendering coordinate system is displayed in combination with the visible range represented by the camera aperture, and the current location of the virtual character is marked in the display interface to attract the user's attention.

[0112] For example, Figure 6 As shown, the solid tetrahedron represents a real object, the dotted man represents a virtual character, the movable range of the virtual character can be represented by a solid color filled area, and the current position of the virtual character is guided by a direction icon.

[0113] In some embodiments, there may be no real object placed in the real scene where the rendering terminal is located, that is, no real object is detected by the target detection algorithm. In this case, the method flow of the rendering terminal locating the virtual character can be found in Figure 7 , mainly includes the following steps:

[0114] S701: Collect the environment image of the real scene and establish a rendering coordinate system.

[0115] This step is described in detail in Figure 5A and Figure 5B , which will not be repeated here.

[0116] S702: When it is determined based on the environment image that there is no real object in the real scene, a prompt message indicating that there is no real object is sent to the acquisition terminal, so that the acquisition terminal displays the movable range of the virtual character of the target object in the acquisition coordinate system according to the prompt message, and synchronizes the movable range to the rendering coordinate system.

[0117] In an embodiment of the present application, when there are no real objects in the real scene where the rendering terminal is located, there is no need to consider the occlusion problem between the virtual character and the real object. At this time, the rendering terminal sends a prompt message that there is no real object to the acquisition terminal. After receiving the prompt message, the acquisition terminal sets and displays the movable range of the virtual character in the acquisition coordinate system according to the visible range of the camera configured for the rendering terminal, and synchronizes the movable range to the rendering coordinate system to prevent the virtual character displayed by the rendering terminal from disappearing from the user's field of view during the interaction.

[0118] S703: Rendering the received three-dimensional reconstruction data of the virtual character within a movable range of the rendering coordinate system, and superimposing the rendered virtual character on the environment image.

[0119] In S703, the rendering terminal receives the 3D reconstruction data of the virtual character sent by the acquisition terminal, and renders the virtual character and displays it superimposed on the environmental image of the real scene according to the design of the AR application. During the display process, the movable range of the virtual character in the rendering coordinate system is set by the acquisition terminal according to the visible range of the camera of the rendering terminal and synchronized to the rendering terminal. Therefore, it can be ensured that when the virtual character moves within the movable range, it will not exceed the real scene, thereby improving the authenticity of the virtual-real fusion.

[0120] The above process is the positioning method flow of the virtual character in the interactive system implemented on the rendering terminal side, corresponding to the rendering terminal. Figure 8 The flowchart of the method for positioning a virtual character in an interactive system implemented by the acquisition terminal side provided in the embodiment of the present application mainly includes the following steps:

[0121] S801: Receive a set of attribute information of a real object sent by a rendering terminal in an interactive system.

[0122] Among them, the attribute information set is extracted by the rendering terminal after determining that there are real objects in the real scene based on the collected environmental image of the real scene. The extraction process and content of the attribute information refer to the description on the rendering terminal side, which will not be repeated here.

[0123] S802: According to the attribute information set, the initial position of the virtual character of the target object in the acquisition coordinate system is adjusted, and the adjusted target position is synchronized to the rendering coordinate system.

[0124] In S802, the initial position of the virtual character in the acquisition coordinate system is adjusted in different ways according to different contents in the attribute information set.

[0125] When the attribute information set includes at least a partial image of the real object and the distance between the real object and the rendering terminal, the initial position of the virtual character in the acquisition coordinate system can be adjusted with the help of the computing power of the transmission terminal. For the specific adjustment process, see Fig. 9A , mainly includes the following steps:

[0126] S802_11: Receive three-dimensional reconstruction data of a virtual object corresponding to the real object, where the virtual object is reconstructed by the transmission terminal based on a partial image of the real object sent by the rendering terminal.

[0127] After the rendering terminal sends the attribute information set to the acquisition terminal through the transmission terminal, the transmission terminal reconstructs the three-dimensional model of the real object based on the local image of the real object with the help of the computing power of the transmission terminal, obtains the virtual object, and sends the three-dimensional reconstruction data of the virtual object (including geometry and texture data) to the acquisition terminal.

[0128] S802_12: Render the three-dimensional reconstruction data of the virtual object in the acquisition coordinate system, and place and display the virtual object according to the distance between the real object and the rendering terminal.

[0129] After receiving the three-dimensional reconstruction data of the virtual object, the acquisition terminal renders the virtual object and places and displays the virtual object according to the distance between the real object and the rendering terminal.

[0130] S802_13: According to the position of the virtual object, adjust the initial position of the virtual character of the target object in the acquisition coordinate system.

[0131] By wearing AR glasses, the collectors on the collection terminal side can see the virtual objects and refer to the distance between the virtual objects and the virtual characters. By changing the camera height, the position of the target object and other operations, they can adjust the initial position of the virtual characters in the collection coordinate system, obtain the target position of the adjusted virtual characters, and synchronize the target position to the rendering coordinate system to ensure that the rendering terminal can correctly place the virtual characters when superimposing and displaying them, thereby improving the authenticity of virtual fusion.

[0132] because Fig. 9A The adjustment process shown requires 3D reconstruction of the real object, which requires high computing power and may cause delays in poor network conditions, causing freezes in the interactive screen and affecting user experience. To solve the above problems, the semantic information of the real object can be used for adjustment.

[0133] When the attribute information set includes at least the semantic information of the real object and the distance between the real object and the rendering terminal, the process of adjusting the initial position of the virtual character in the acquisition coordinate system is as follows: Fig. 9B , mainly includes the following steps:

[0134] S802_21: According to the semantic information of the real object, a virtual object corresponding to the semantic information is obtained from a preset database, and the virtual object is displayed in the acquisition coordinate system according to the distance between the real object and the rendering terminal.

[0135] In the embodiment of the present application, the preset database contains three-dimensional models of various types of objects, so virtual objects corresponding to real objects of corresponding categories can be obtained from the preset database.

[0136] Taking the semantic information as the category of real objects as an example, assuming that the real object in the real scene of the rendering terminal is a sofa, after receiving the semantic information, the collection terminal obtains the virtual object of the sofa from the preset database, and places the virtual object according to the distance between the sofa and the rendering terminal in the real scene to provide a reference for the positioning of the virtual character. The collection personnel can see the virtual object of the sofa by wearing AR glasses on the side of the collection terminal.

[0137] S802_22: According to the position of the virtual object, adjust the initial position of the virtual character of the target object in the acquisition coordinate system.

[0138] Still taking the virtual object as an example, after placing the virtual object of the sofa, refer to the distance between the virtual object of the sofa and the virtual character, adjust the initial position of the virtual character in the acquisition coordinate system by changing the camera height, the position of the target object and other operations, obtain the target position of the adjusted virtual character, and synchronize the target position to the rendering coordinate system to ensure that the rendering terminal can correctly place the virtual character when superimposing and displaying the virtual character, thereby improving the authenticity of virtual fusion.

[0139] Relative to Fig. 9A The adjustment method shown is Fig. 9B It is more real-time and can be applied to various interactive scenarios.

[0140] S803: Sending the 3D reconstruction data of the virtual character of the target object, so that the rendering terminal renders the 3D reconstruction data at the target position of the rendering coordinate system, and superimposes and displays the rendered virtual character in the environment image.

[0141] In the embodiments of the present application, the three-dimensional reconstruction of the virtual character involves human body shape reconstruction, motion reconstruction, and material reconstruction. In terms of human body shape reconstruction, a semantic layered human body expression, constraint, and solution method based on a parametric human body model and human body semantic segmentation is used to reconstruct the human body geometry model. In this way, while improving the accuracy of three-dimensional reconstruction of the human body, multi-layer semantic reconstruction of the human body's dynamic three-dimensional information is achieved. In terms of human body motion reconstruction, a joint motion tracking method based on skeleton tracking and a parametric human body model is used to achieve a joint solution of human skeleton motion and surface non-rigid motion, which can improve the accuracy of human motion capture; and, in view of the fact that the motion capture method under single viewpoint conditions cannot handle the complex difficulties of rapid motion and human self-occlusion, the embodiments of the present application use a semantic-based bidirectional hybrid motion capture algorithm to improve the robustness of human motion capture under complex conditions.

[0142] Among them, the relevant technologies of real-time 3D reconstruction of virtual characters can refer to Doublefusion, Function4D, and PifuHD, etc. Since the 3D reconstruction process of virtual characters is not the focus of this application, it will not be described in detail here.

[0143] In an optional implementation, when S803 is executed, after the camera and host in the acquisition terminal complete the three-dimensional reconstruction of the virtual character, the three-dimensional reconstruction data of the virtual character (including geometry and texture data) is sent to the rendering terminal through the transmission terminal, and the transmission terminal compresses the three-dimensional reconstruction data and distributes it to the rendering terminal. The rendering terminal renders at the target position of the rendering coordinate system according to the three-dimensional reconstruction data, and superimposes the rendered virtual character on the environmental image. Since the target position of the virtual character is adjusted according to the attribute information of the real object in the real scene, there will be no occlusion with the real object when superimposed, which improves the authenticity of the fusion of virtual and real, and thus enhances the interactive experience.

[0144] In another optional implementation, the acquisition terminal may send the 3D reconstruction data of the virtual character to the rendering terminal, and the rendering terminal may pre-generate the virtual character based on the 3D reconstruction data. In this way, during the interaction process, the acquisition terminal may send the geometric data reflecting the posture of the target object in the 3D reconstruction data to the rendering terminal to reduce the amount of data transmitted and improve the real-time nature of the interaction. In this way, the rendering terminal may drive the virtual character to be displayed to move based on the received geometric data, and obtain the virtual character with a real posture for display.

[0145] In some embodiments, there may be no real object placed in the real scene where the rendering terminal is located, that is, no real object is detected by the target detection algorithm. In this case, the method flow of the acquisition terminal locating the virtual character can be found in Fig.10 , mainly includes the following steps:

[0146] S1001: Receive a set of attribute information of a real object sent by a rendering terminal in an interactive system.

[0147] S1002: Receive prompt information of no real object sent by the rendering terminal when determining that no real object exists in the real scene according to the environment image.

[0148] In an embodiment of the present application, when there is no real object in the real scene where the rendering terminal is located, there is no need to consider the occlusion problem between the virtual character and the real object. At this time, the rendering terminal sends a prompt message that there is no real object to the acquisition terminal.

[0149] S1003: According to the prompt information, the movable range of the target object in the acquisition coordinate system is displayed, and the movable range is synchronized to the rendering coordinate system, so that the rendering terminal renders within the movable range of the rendering coordinate system according to the three-dimensional reconstruction data of the virtual character, and overlays the rendered virtual character on the environmental image.

[0150] In S1003, after receiving the prompt information, the acquisition terminal sets and displays the movable range of the virtual character in the acquisition coordinate system according to the visible range of the camera configured by the rendering terminal, and synchronizes the movable range to the rendering coordinate system to ensure that the virtual character does not exceed the real scene when moving within the movable range, thereby improving the authenticity of the virtual-real fusion.

[0151] In the embodiments of the present application, the acquisition terminal is not only responsible for reconstructing the virtual character and sending the three-dimensional reconstruction data of the virtual character, and the rendering terminal is not only responsible for superimposing and displaying the virtual character. The acquisition terminal adjusts the initial position of the virtual character according to the environmental information of the real scene fed back by the rendering terminal, and synchronizes it to the rendering terminal. In this way, through the two-way feedback of the rendering terminal and the acquisition terminal, the interactivity between the virtual character and the objects in the real scene is enhanced, and abnormal phenomena such as occlusion of the virtual character and the real object and exceeding the real scene can be reduced. To a certain extent, the sense of disharmony of the fusion of the virtual character and the real scene is weakened, the authenticity of the fusion of the virtual and the real is improved, and the user's immersive experience is thereby improved.

[0152] See also Fig.11 , is a schematic diagram of the interaction scene of the acquisition terminal and the rendering terminal provided in the embodiment of the present application. Among them, the acquisition terminal and the rendering terminal are respectively located in two different scenes. For the purpose of distinguishing the descriptions, the scene where the acquisition terminal is located is recorded as scene 1, and the scene where the rendering terminal is located is recorded as scene 2.

[0153] In scene 1, the host in the acquisition terminal uses the geometric data and texture data of ActorA collected by the camera to reconstruct the 3D human body model of ActorA, and transmits the 3D reconstruction data of ActorA's 3D human body model to the rendering terminal in scene 2 through the cloud.

[0154] In scene 2, the observer wears a rendering terminal, which can be AR glasses or an AR helmet, and is equipped with a camera to collect environmental images of the real scene. Real objects A and B are placed in scene 2. By performing semantic segmentation and distance analysis on the environmental images containing real objects A and B, the semantic information of real objects A and B, as well as their respective distances from the rendering terminal, are extracted and transmitted to the collection terminal in scene 1 through the cloud.

[0155] In scene 1, the collector on the collection terminal side wears AR glasses. After the collection terminal receives the semantic information of real object A and real object B sent by the rendering terminal, as well as the distance between each object and the rendering terminal, the collector can see the virtual object A and virtual object B matched from the preset database based on the semantic information through the AR glasses he wears, and obtain the positions of virtual object A and virtual object B according to the distance between real object A and real object B and the rendering terminal, so as to know the real scene in scene 2. In this way, the collection terminal adjusts the original position of the virtual character in the collection coordinate system through the distance between virtual object A and virtual object B and the reconstructed virtual character, prevents the virtual character from being blocked by real object A and real object B, and synchronizes the adjusted target position to the rendering terminal in scene 2.

[0156] In scene 2, after receiving the 3D reconstruction data sent by the acquisition terminal, the rendering terminal renders and displays the virtual character at the target position of the rendering coordinate system. The observer can see the AR effect after the virtual character and the real scene are superimposed and displayed through the rendering terminal.

[0157] based on Fig.11 The interactive scene shown, Fig.12 The flowchart of the interactive method for positioning a virtual character provided in the embodiment of the present application mainly includes the following steps:

[0158] S1201: The rendering terminal receives the three-dimensional reconstruction data of the virtual character sent by the acquisition terminal.

[0159] S1202: The rendering terminal collects an environmental image of a real scene.

[0160] S1203: The rendering terminal determines whether the environment image contains the target plane. If so, execute S1204; otherwise, return to S1202 to re-capture the environment image.

[0161] S1204: The rendering terminal establishes a rendering coordinate system using the detected target plane as the ground.

[0162] S1205: The rendering terminal detects whether there is a real object in the real scene based on the environment image. If so, execute S1206; otherwise, execute S1211.

[0163] S1206: The rendering terminal performs semantic segmentation and distance analysis on the environment image to extract semantic information of the real object and the distance between the real object and the rendering terminal.

[0164] S1207: The rendering terminal sends the semantic information of the real object and the distance between the real object and the rendering terminal to the acquisition terminal.

[0165] S1208: The collection terminal renders the virtual object according to the semantic information of the real object and the distance between the real object and the rendering terminal, and displays it to the collection personnel through the AR glasses, so that the collection personnel can adjust the initial position of the virtual character in the collection coordinate system with reference to the virtual object.

[0166] S1209: The acquisition terminal synchronizes the adjusted target position to the rendering coordinate system.

[0167] S1210: The rendering terminal renders the three-dimensional reconstruction data at a target position in a rendering coordinate system, and superimposes the rendered virtual character in the environment image.

[0168] S1211: The rendering terminal sends a prompt message indicating that there is no real object to the acquisition terminal.

[0169] S1212: The acquisition terminal displays the movable range of the virtual character in the acquisition coordinate system according to the prompt information, and synchronizes the movable range to the rendering coordinate system.

[0170] S1213: The rendering terminal renders the three-dimensional reconstruction data within the movable range of the rendering coordinate system, and overlays the rendered virtual character on the environment image.

[0171] S1214: The rendering terminal displays an indicator of the virtual character's current location on the user interface to enhance the attention of the virtual character.

[0172] In a method for positioning a virtual character in an interactive system provided in an embodiment of the present application, a rendering terminal establishes a rendering coordinate system based on a captured environmental image of a real scene, extracts semantic information of real objects in the real scene and the distance between the real objects and the rendering terminal, and sends the information to the acquisition terminal; the acquisition terminal adjusts the initial position of the virtual character in the acquisition coordinate system based on the received semantic information of the real objects and the distance between the real objects and the rendering terminal, and synchronizes the adjusted target position to the rendering coordinate system. In this way, the accuracy of the positioning of the virtual character is improved through the two-way feedback mechanism of the acquisition terminal and the rendering terminal in the interactive system. After the rendering terminal receives the three-dimensional reconstruction data of the virtual character, the virtual character is rendered at the target position in the rendering coordinate system and superimposed on the environmental image, thereby achieving accurate fusion of the virtual character and the real scene, reducing abnormal phenomena such as occlusion or floating of the virtual character and the real object, and to a certain extent weakening the sense of disharmony of the fusion of the virtual character and the real scene, thereby improving the authenticity of the fusion of the virtual and the real.

[0173] Based on the same technical concept, an embodiment of the present application provides a rendering terminal, which can implement the steps of the positioning method of the virtual character in the interactive system in the above embodiment and achieve the same technical effect.

[0174] See also Fig.13 , the rendering terminal includes a camera 1301, a processor 1302, a memory 1303, a display screen 1304 and a communication interface 1305, wherein the camera 1301, the communication interface 1305, the display screen 1304, the memory 1303 and the processor 1302 are connected via a bus 1306;

[0175] The memory 1303 stores a computer program, and the processor 1302 performs the following operations according to the computer program:

[0176] The camera 1301 is used to collect an environmental image of a real scene and establish a rendering coordinate system;

[0177] When it is determined according to the environment image that a real object exists in the real scene, extracting a set of attribute information of the real object;

[0178] Sending the attribute information set to the acquisition terminal in the interactive system through the communication interface 1305, so that the acquisition terminal adjusts the initial position of the virtual character of the target object in the acquisition coordinate system according to the attribute information set, and synchronizes the adjusted target position to the rendering coordinate system;

[0179] The three-dimensional reconstruction data of the virtual character sent by the acquisition terminal is received through the communication interface 1305, the three-dimensional reconstruction data is rendered at the target position of the rendering coordinate system, and the rendered virtual character is superimposed and displayed in the environment image through the display screen 1304.

[0180] Optionally, when it is determined according to the environment image that there is no real object in the real scene, the processor 1302 further executes:

[0181] Sending prompt information indicating that there is no real object to the acquisition terminal through the communication interface 1305, so that the acquisition terminal displays the movable range of the virtual character of the target object in the acquisition coordinate system according to the prompt information, and synchronizes the movable range to the rendering coordinate system;

[0182] The three-dimensional reconstruction data of the virtual character sent by the acquisition terminal is received through the communication interface 1305, the three-dimensional reconstruction data is rendered within the movable range of the rendering coordinate system, and the rendered virtual character is superimposed and displayed in the environment image through the display screen 1304.

[0183] Optionally, the processor 1302 establishes a rendering coordinate system, and the specific operations are:

[0184] Extracting three-dimensional point cloud data contained in the environment image;

[0185] Detecting, based on the three-dimensional point cloud data, whether the environment image contains at least one plane;

[0186] If included, a target plane is selected as the ground from at least one detected plane, and the rendering coordinate system is established; otherwise, the environment image is recaptured.

[0187] Optionally, the attribute information set includes at least a distance between the real object and the rendering terminal, and semantic information of the real object; or

[0188] The attribute information set includes at least the distance between the real object and the rendering terminal, and a partial image of the real object.

[0189] Based on the same technical concept, an embodiment of the present application provides a collection terminal, which can implement the steps of the positioning method of the virtual character in the interactive system in the above embodiment and achieve the same technical effect.

[0190] See also Fig.14The acquisition terminal includes a camera 1401 and a host 1402, wherein the host 1402 includes a processor 1402_1, a memory 1402_2, a display screen 1402_3 and a communication interface 1402_4, and the camera 1401, the communication interface 1402_4, the display screen 1402_3, the memory 1402_2 and the processor 1402_1 are connected via a bus 1403;

[0191] The display screen 1402_3 is used to display the reconstructed virtual character;

[0192] The memory 1402_2 stores a computer program, and the processor 1402_1 performs the following operations according to the computer program:

[0193] Receiving, through the communication interface 1402_4, a set of attribute information of a real object sent by a rendering terminal in the interactive system, the set of attribute information being extracted by the rendering terminal after determining that a real object exists in the real scene according to a captured environment image of the real scene;

[0194] According to the attribute information set, adjusting the initial position of the virtual character of the target object in the acquisition coordinate system, and synchronizing the adjusted target position to the rendering coordinate system, wherein the rendering coordinate system is established by the rendering terminal according to the environment image;

[0195] The three-dimensional reconstruction data of the virtual character of the target object is sent through the communication interface 1402_4, so that the rendering terminal renders at the target position of the rendering coordinate system according to the three-dimensional reconstruction data, and superimposes the rendered virtual character on the environmental image, wherein the virtual character is reconstructed according to the data collected by the camera.

[0196] Optionally, the attribute information set includes at least the distance between the real object and the rendering terminal, and semantic information of the real object;

[0197] The processor 1402_1 adjusts the initial position of the virtual character of the target object in the acquisition coordinate system according to the attribute information set, and the specific operation is:

[0198] According to the semantic information of the real object, a virtual object corresponding to the semantic information is acquired from a preset database, and according to the distance between the real object and the rendering terminal, the virtual object is displayed in the acquisition coordinate system;

[0199] According to the position of the virtual object, the initial position of the virtual character of the target object in the acquisition coordinate system is adjusted.

[0200] Optionally, the attribute information set includes at least a distance between the real object and the rendering terminal, and a partial image of the real object;

[0201] The processor 1402_1 adjusts the initial position of the virtual character of the target object in the acquisition coordinate system according to the attribute information set, and the specific operation is:

[0202] Receiving, via the communication interface 1402_4, 3D reconstruction data of a virtual object corresponding to the real object, the virtual object being reconstructed by the transmission terminal according to the partial image of the real object sent by the rendering terminal;

[0203] Rendering the three-dimensional reconstruction data of the virtual object in the acquisition coordinate system, and placing and displaying the virtual object according to the distance between the real object and the rendering terminal;

[0204] According to the position of the virtual object, the initial position of the virtual character of the target object in the acquisition coordinate system is adjusted.

[0205] Optionally, the processor 1402_1 further executes:

[0206] Receiving, through the communication interface 1402_4, prompt information indicating that there is no real object, which is sent by the rendering terminal when determining, according to the environment image, that there is no real object in the real scene;

[0207] According to the prompt information, the movable range of the target object in the acquisition coordinate system is displayed through the display screen 1402_3, and the movable range is synchronized to the rendering coordinate system, so that the rendering terminal renders within the movable range of the rendering coordinate system according to the three-dimensional reconstruction data, and superimposes the rendered virtual character in the environmental image.

[0208] It should be noted that Fig.13 , 14 This is only an example, and the necessary hardware for the rendering terminal and the acquisition terminal to execute the steps of the method for positioning a virtual character in an interactive system provided in an embodiment of the present application is provided. Not shown, the rendering terminal and the acquisition terminal also include common hardware of a display device, such as a speaker, a microphone, etc.

[0209] Embodiments of the present application Fig.13 , 14The processor involved may be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0210] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the method of the aforementioned embodiment.

[0211] The embodiments of the present application also provide a computer program product for storing a computer program, wherein the computer program is used to execute the method of the aforementioned embodiment.

[0212] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0213] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0214] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0215] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0216] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for positioning a virtual character, characterized in that: Applied to a first rendering terminal, the first rendering terminal is any rendering terminal in an interactive system for realizing a remote conference, the any rendering terminal is a head mounted display device, and the any rendering terminal includes a camera, the interactive system further includes a first acquisition terminal at one end of the first rendering terminal and a second rendering terminal and a second acquisition terminal at the opposite end of the one end, the any acquisition terminal in the interactive system is used to obtain three-dimensional reconstruction data of a user of a rendering terminal at the same end of the head mounted display device and the any acquisition terminal, including: Capturing an environmental image of a real scene through a camera of the first rendering terminal to establish a rendering coordinate system; When it is determined according to the environmental image that there is a real object in the real scene, extract the attribute information set of the real object; send the attribute information set to the second acquisition terminal, so that the second acquisition terminal adjusts the initial position of the virtual character of the target object corresponding to the second acquisition terminal in the acquisition coordinate system of the second acquisition terminal according to the attribute information set, and synchronizes the adjusted target position to the rendering coordinate system of the first rendering terminal; receive the three-dimensional reconstruction data of the virtual character sent by the second acquisition terminal, render at the target position of the rendering coordinate system according to the received three-dimensional reconstruction data of the virtual character, and overlay and display the rendered virtual character in the environmental image, wherein the attribute information set includes at least the distance between the real object and the first rendering terminal, and the semantic information of the real object; or the attribute information set includes at least the distance between the real object and the first rendering terminal, and the local image of the real object; When it is determined according to the environmental image that there are no real objects in the real scene, a prompt message indicating that there are no real objects is sent to the second acquisition terminal, so that the second acquisition terminal responds to the prompt message, sets the movable range of the virtual character of the target object corresponding to the second acquisition terminal within the acquisition coordinate system of the second acquisition terminal according to the visible range of the second acquisition terminal, and synchronizes the movable range to the rendering coordinate system of the first rendering terminal; and receives the three-dimensional reconstruction data of the virtual character sent by the second acquisition terminal, renders within the movable range of the rendering coordinate system according to the received three-dimensional reconstruction data of the virtual character, and superimposes and displays the rendered virtual character in the environmental image.

2. The method according to claim 1, characterized in that The establishing of the rendering coordinate system comprises: Extracting three-dimensional point cloud data contained in the environment image; Detecting, based on the three-dimensional point cloud data, whether the environment image contains at least one plane; If included, a target plane is selected as the ground from at least one detected plane, and the rendering coordinate system is established; otherwise, the environment image is recaptured.

3. The method according to claim 2, characterized in that The detecting, based on the three-dimensional point cloud data, whether the environment image includes at least one plane comprises: According to the 3D point cloud data, multiple planes are randomly fitted; For each plane, determine the first number of points in the three-dimensional point cloud data whose distance to the plane exceeds a preset distance threshold, and if the first number of points is greater than a set first number threshold, discard the plane; if the first number of points is not greater than the first number threshold, retain the plane; from the retained planes, select a plane with the least number of points exceeding the preset distance threshold, and use the selected plane as the detected target plane; or, For each plane, determine the second number of points contained in the plane whose distance is less than a preset distance threshold; if the second number of points is greater than a set second number threshold, retain the plane; if the second number of points is not greater than the second number threshold, discard the plane; from the retained planes, select the plane with the least number of points less than the preset distance threshold, and determine the selected plane as the target plane.

4. A method for positioning a virtual character, characterized in that: A second acquisition terminal applied to an interactive system, the second acquisition terminal being any acquisition terminal in the interactive system for realizing a remote conference, the any acquisition terminal being used to obtain three-dimensional reconstruction data of a user of a rendering terminal at the same end as a head mounted display device and the any acquisition terminal, the interactive system further comprising a second rendering terminal at one end with the second acquisition terminal, and a first rendering terminal and a first acquisition terminal at the opposite end of the one end, including: Receive an attribute information set of a real object sent by a first rendering terminal in an interactive system; adjust the initial position of a virtual character of a target object corresponding to the second acquisition terminal in an acquisition coordinate system corresponding to the second acquisition terminal according to the attribute information set, and synchronize the adjusted target position to the rendering coordinate system; send three-dimensional reconstruction data of the virtual character of the target object, so that the first rendering terminal performs rendering at the target position of the rendering coordinate system according to the three-dimensional reconstruction data, and overlays and displays the rendered virtual character in an environmental image, wherein the attribute information set includes at least the distance between the real object and the first rendering terminal, and the semantic information of the real object; or the attribute information set includes at least the distance between the real object and the first rendering terminal, and a partial image of the real object; Receive prompt information of no real object sent by the first rendering terminal when determining that no real object exists in the real scene according to the environmental image; in response to the prompt information, set the movable range of the target object corresponding to the second acquisition terminal in the acquisition coordinate system of the second acquisition terminal according to the visible range of the second acquisition terminal, synchronize the movable range to the rendering coordinate system of the first rendering terminal, and send the three-dimensional reconstruction data of the virtual character, so that the first rendering terminal renders within the movable range of the rendering coordinate system according to the received three-dimensional reconstruction data of the virtual character, and superimposes the rendered virtual character on the environmental image.

5. The method according to claim 4, characterized in that When the attribute information set includes at least the distance between the real object and the first rendering terminal, and the semantic information of the real object; The adjusting, according to the attribute information set, an initial position of a virtual character of a target object corresponding to the second acquisition terminal in a collection coordinate system corresponding to the second acquisition terminal includes: According to the semantic information of the real object, a virtual object corresponding to the semantic information is acquired from a preset database, and according to the distance between the real object and the first rendering terminal, the virtual object is displayed in a collection coordinate system corresponding to the second collection terminal; According to the position of the virtual object, an initial position of the virtual character of the target object corresponding to the second acquisition terminal in the acquisition coordinate system corresponding to the second acquisition terminal is adjusted.

6. The method according to claim 4, characterized in that When the attribute information set includes at least a distance between the real object and the first rendering terminal, and a partial image of the real object; The adjusting, according to the attribute information set, an initial position of a virtual character of a target object corresponding to the second acquisition terminal in a collection coordinate system corresponding to the second acquisition terminal includes: receiving three-dimensional reconstruction data of a virtual object corresponding to the real object, wherein the virtual object is reconstructed by a transmission terminal according to a partial image of the real object sent by the first rendering terminal; Rendering the three-dimensional reconstruction data of the virtual object in a collection coordinate system corresponding to the second collection terminal, and placing and displaying the virtual object according to the distance between the real object and the first rendering terminal; According to the position of the virtual object, an initial position of the virtual character of the target object corresponding to the second acquisition terminal in the acquisition coordinate system corresponding to the second acquisition terminal is adjusted.

7. A first rendering terminal, characterized in that: An interactive system for implementing a remote conference, the interactive system further comprising a first acquisition terminal at one end with the first rendering terminal, and a second rendering terminal and a second acquisition terminal at the opposite end of the one end, any one of the acquisition terminals in the interactive system is used to obtain three-dimensional reconstruction data of a user of a rendering terminal at the same end as a head-mounted display device and any one of the acquisition terminals, any one of the rendering terminals in the interactive system is a head-mounted display device, the first rendering terminal comprises a camera, a processor, a memory, a display screen and a communication interface, the camera, the memory, the display screen, the communication interface and the processor are connected via a bus: The memory stores a computer program, and the processor performs the following operations according to the computer program: The camera is used to collect an environmental image of a real scene and establish a rendering coordinate system; When it is determined according to the environment image that a real object exists in the real scene, extracting a set of attribute information of the real object; The attribute information set is sent to the second acquisition terminal through the communication interface, so that the second acquisition terminal adjusts the initial position of the virtual character of the target object corresponding to the second acquisition terminal in the acquisition coordinate system of the second acquisition terminal according to the attribute information set, and synchronizes the adjusted target position to the rendering coordinate system of the first rendering terminal; the three-dimensional reconstruction data of the virtual character sent by the second acquisition terminal is received through the communication interface, and rendering is performed at the target position of the rendering coordinate system according to the received three-dimensional reconstruction data of the virtual character, and the rendered virtual character is superimposed and displayed in the environment image through the display screen, wherein the attribute information set includes at least the distance between the real object and the first rendering terminal, and the semantic information of the real object; or the attribute information set includes at least the distance between the real object and the first rendering terminal, and the local image of the real object; When it is determined according to the environmental image that there are no real objects in the real scene, a prompt message indicating that there are no real objects is sent to the second acquisition terminal, so that the second acquisition terminal responds to the prompt message, sets the movable range of the virtual character of the target object corresponding to the second acquisition terminal within the acquisition coordinate system of the second acquisition terminal according to the visible range of the second acquisition terminal, and synchronizes the movable range to the rendering coordinate system of the first rendering terminal; and receives the three-dimensional reconstruction data of the virtual character sent by the second acquisition terminal, renders within the movable range of the rendering coordinate system according to the received three-dimensional reconstruction data of the virtual character, and superimposes and displays the rendered virtual character in the environmental image.

8. A second acquisition terminal, characterized in that: An interactive system for implementing a remote conference, the interactive system further comprising a second rendering terminal at one end with the second acquisition terminal, and a first rendering terminal and a first acquisition terminal at the opposite end of the one end, any one of the acquisition terminals in the interactive system is used to obtain three-dimensional reconstruction data of a user of a rendering terminal at the same end as a head mounted display device and any one of the acquisition terminals, and the second acquisition terminal comprises a camera and a host, the host comprises a processor, a memory, a display screen and a communication interface, the camera, the memory, the display screen, the communication interface and the processor are connected via a bus: The display screen is used to display the reconstructed virtual character; The memory stores a computer program, and the processor performs the following operations according to the computer program: Receiving, via the communication interface, a set of attribute information of a real object sent by a first rendering terminal in the interactive system; According to the attribute information set, adjust the initial position of the virtual character of the target object corresponding to the second acquisition terminal in the acquisition coordinate system corresponding to the second acquisition terminal, and synchronize the adjusted target position to the rendering coordinate system; send the three-dimensional reconstruction data of the virtual character of the target object through the communication interface, so that the first rendering terminal renders at the target position of the rendering coordinate system according to the three-dimensional reconstruction data, and superimposes and displays the rendered virtual character in the environment image, wherein the virtual character is reconstructed according to the data collected by the camera, and the attribute information set includes at least the distance between the real object and the first rendering terminal, and the semantic information of the real object; Or the attribute information set includes at least the distance between the real object and the first rendering terminal, and a partial image of the real object; receiving prompt information indicating that there is no real object, which is sent by the first rendering terminal when determining that there is no real object in the real scene according to the environment image; In response to the prompt information, according to the visible range of the second acquisition terminal, the movable range of the target object corresponding to the second acquisition terminal is set within the acquisition coordinate system of the second acquisition terminal, and the movable range is synchronized to the rendering coordinate system of the first rendering terminal, and the three-dimensional reconstruction data of the virtual character is sent, so that the first rendering terminal renders within the movable range of the rendering coordinate system according to the received three-dimensional reconstruction data of the virtual character, and superimposes the rendered virtual character in the environmental image.

Images