Method and device for driving virtual characters based on visual semantics

Abstract

The present application relates to the field of AR technology, and provides a method and device for driving a virtual character based on visual semantics, wherein a three-dimensional space coordinate system is established by using a real scene image, and semantic information, distance information and scale information of a real object are extracted, the virtual character is placed in the three-dimensional space coordinate system according to the distance information, and the size of the virtual character is adjusted according to the scale information so that the proportion of the virtual character and the real object are adapted, and after the virtual character is placed, the virtual character is driven to generate corresponding behavior of the real object according to target driving data obtained by using the semantic information of the real object, so as to realize automatic interaction between the virtual character and the real object, so that the virtual character is better integrated into the real scene, and the sense of disharmony of the virtual object in the real scene is reduced to a certain extent, and the authenticity of the fusion of virtual and real is improved, thereby enhancing the AR experience of the user.

Description

Technical Field

[0001] The present application relates to the field of augmented reality (AR) technology, and in particular to a method and device for driving a virtual character based on visual semantics. Background Art

[0002] AR technology is a new technology developed on the basis of Virtual Reality (VR). It enhances the real scene by superimposing generated virtual objects on the real scene, and has been widely used in all walks of life.

[0003] At present, when a virtual character is superimposed on a real scene, after the virtual character is placed, the virtual character is generally driven to move in the real scene directly according to a preset driving method (such as real-time skeleton driving, voice driving, face capture expression driving, etc.), or the virtual character is driven to move in the real scene using pre-stored driving data. Because the virtual character does not interact with the real scene and objects in the real scene, that is, the virtual character has no interactive association with the real environment and objects, the virtual character cannot be truly integrated into the real scene, thereby reducing the user's AR experience. Summary of the invention

[0004] The embodiments of the present application provide a method and device for driving a virtual character based on visual semantics, which are used to improve the authenticity of virtual-real fusion through the interaction between the virtual character and objects in the real scene.

[0005] On the one hand, an embodiment of the present application provides a method for driving a virtual character based on visual semantics, which is applied to an AR device, and the method includes:

[0006] Acquire a real scene image, and establish a three-dimensional space coordinate system according to the real scene image;

[0007] When a real object is detected according to the real scene image, semantic information, distance information and scale information of the real object are extracted;

[0008] Acquiring target driving data according to the semantic information of the real object;

[0009] placing the virtual character in the three-dimensional space coordinate system according to the distance information, and adjusting the size of the virtual character according to the scale information so that the virtual character fits the real object;

[0010] The target driving data is used to drive the virtual character to produce a behavior corresponding to the real object.

[0011] On the other hand, an embodiment of the present application provides an AR device, including a multi-eye camera, a processor, a memory, and a display screen, wherein the display screen, the memory, the multi-eye camera, and the processor are connected via a bus:

[0012] The multi-eye camera is used to collect real scene images;

[0013] The memory includes a data storage unit and a program storage unit, the program storage unit stores a computer program, and the processor performs the following operations according to the computer program:

[0014] Acquire a real scene image, display it on the display screen, and establish a three-dimensional space coordinate system based on the real scene image;

[0015] When a real object is detected according to the real scene image, semantic information, distance information and scale information of the real object are extracted;

[0016] Acquire target driving data according to the semantic information of the real object, and store it in the data storage unit;

[0017] placing the virtual character in the three-dimensional space coordinate system according to the distance information, and adjusting the size of the virtual character according to the scale information so that the virtual character fits the real object;

[0018] The target driving data is used to drive the virtual character to generate a behavior corresponding to the real object, and the driven virtual character is displayed by overlaying on the display screen.

[0019] Optionally, when multiple real objects are detected, the processor acquires target drive data according to semantic information of the real objects, and the specific operation is:

[0020] Sort the importance of multiple real objects according to their semantic information;

[0021] The target driving data corresponding to the semantic information of each real object is obtained in sequence from high to low importance.

[0022] Optionally, the processor drives the virtual character to generate a behavior corresponding to the real object by using the target driving data, and the specific operation is:

[0023] According to the acquisition order of the target driving data, the corresponding virtual characters are driven in sequence to produce corresponding behaviors corresponding to the real objects.

[0024] Optionally, when there are multiple virtual characters, when the processor places the virtual characters in the three-dimensional space coordinate system according to the distance information, the processor further executes:

[0025] Determining the position of each virtual character's foot in the three-dimensional space coordinate system in sequence according to the distance information;

[0026] Move each virtual character according to a preset threshold value to determine the activity range of the multiple virtual characters in the three-dimensional space coordinate system;

[0027] The activity range is equally divided according to the number of the virtual characters to obtain an activity sub-range for each virtual character.

[0028] Optionally, when a real object is detected, the distance information includes the distance from the real object to the AR device;

[0029] When multiple real objects are detected, the distance information includes the distance from each real object to the AR device, and the distance between the real objects.

[0030] Optionally, each real object is associated with one or more virtual characters, and each semantic information corresponds to at least one set of target driving data.

[0031] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the visual semantics-based virtual character driving method provided in the embodiment of the present application.

[0032] In a method and device for driving a virtual character based on visual semantics provided in an embodiment of the present application, a three-dimensional space coordinate system is established by using a real scene image, and semantic information, distance information and scale information of the detected real object are extracted, the virtual character is placed in the three-dimensional space coordinate system according to the distance information, and the size of the virtual character is adjusted according to the scale information, and the virtual character is driven to produce corresponding behavior of the real object according to the target driving data obtained by using the semantic information of the real object. In this way, the interaction between the virtual character and the real object is realized through the semantic information, distance information and scale information, so that the virtual character is better integrated into the real scene, the authenticity of the virtual-real fusion is improved, and the user's AR experience is enhanced.

[0033] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1A A schematic diagram of an AR device provided in an embodiment of the present application;

[0036] Figure 1B A schematic diagram of an indoor navigation application scenario provided in an embodiment of the present application;

[0037] Figure 2 A flow chart of a method for driving a virtual character based on visual semantics provided in an embodiment of the present application;

[0038] Figure 3 A flow chart of a method for establishing a three-dimensional space coordinate system based on a real scene image provided in an embodiment of the present application;

[0039] Figure 4 A distance analysis schematic diagram provided for an embodiment of the present application;

[0040] Figure 5 A flowchart of another method for driving a virtual character based on visual semantics provided in an embodiment of the present application;

[0041] Figure 6 A flow chart for determining the activity range of a virtual character provided in an embodiment of the present application;

[0042] Fig. 7A This is a rendering of a virtual character driven based on visual semantics provided by an embodiment of the present application in a bedroom scene provided by an embodiment of the present application;

[0043] Figure 7B This is a rendering of a virtual character driven based on visual semantics provided by an embodiment of the present application in a living room scene provided by an embodiment of the present application;

[0044] Figure 8 A hardware structure diagram of an AR device provided in an embodiment of the present application;

[0045] Fig. 9 This is a functional structure diagram of the AR device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] At present, the virtual character interaction system of augmented reality is still in the exploratory stage in various industries, and the device form is not unified. In actual AR applications, virtual characters generally perform corresponding movements in specific virtual scenes according to the scene settings.

[0047] Considering the limited computing power of AR devices, high-complexity operations are difficult to run locally on AR devices, and often rely on the computing power of cloud servers for complex operations; and if the posture of the virtual character is calculated in real time, a higher frame rate is required, which increases the transmission capacity of the existing network. Therefore, the current virtual character interaction is usually driven by pre-made voice and / or posture data, where voice data is used to drive the facial expressions of the virtual character, and posture data is used to drive the torso and upper limb movements of the virtual character.

[0048] Most AR devices are equipped with multiple cameras, such as Figure 1A As shown, the camera of the AR device can be used to capture the current real environment, and based on the current real environment, the corresponding virtual character can be customized for "enhancement".

[0049] In AR applications, after locating the position of the virtual character in the real scene using Simultaneous Localization and Mapping (SLAM) technology or other technologies, the virtual character is simply superimposed and displayed in the real scene, driven by pre-made driving data, and combined with a two-dimensional display screen for large-screen display.

[0050] For example, in Figure 1B In the indoor navigation scene shown, the real indoor scene is displayed in real time through a tablet, and arrows and virtual characters are superimposed on the real scene to enhance indoor exit navigation.

[0051] However, currently when virtual characters are superimposed on real scenes, there is a lack of interaction between the virtual characters and the real scenes and real objects, resulting in the virtual characters being unable to truly integrate into the real scenes, while the semantic information of the real objects has no corresponding effect on the movement posture of the virtual characters.

[0052] Among them, the interaction in the embodiment of the present application refers to the ability of the virtual character to express corresponding actions according to the objects in the real scene, which is a further interaction under the condition of preventing the virtual character from floating in the real scene or blocking each other with the real objects through distance judgment.

[0053] In view of this, the embodiments of the present application provide a method and device for driving a virtual character based on visual semantics. According to the real scene image captured by the camera of the AR device, the semantic information and distance information of the real object are extracted. After the virtual character is placed in the real scene according to the distance information, the target driving data obtained according to the semantic information is used to drive the virtual character to produce corresponding behaviors corresponding to the real object, thereby realizing automatic interaction between the virtual character and the real object, reducing the sense of disharmony of the combination of the virtual character and the real scene to a certain extent, improving the authenticity of the fusion of virtual and real, and thus enhancing the user's AR experience.

[0054] See also Figure 2 , is a flow chart of a driving method of a virtual character based on visual semantics provided in an embodiment of the present application. The process is executed by an AR device and mainly includes the following steps:

[0055] S201: Acquire a real scene image, and establish a three-dimensional space coordinate system according to the real scene image.

[0056] When executing S201, after opening the AR application installed on the AR device, the user rotates the body or head to make the multi-eye camera on the AR device scan the surrounding real scene images and transmit them to the processor, and the processor establishes a three-dimensional space coordinate system based on the real scene images.

[0057] The process of establishing the coordinate system can be found in Figure 3 , mainly includes the following steps:

[0058] S2011: Extract three-dimensional point cloud data contained in real scene images.

[0059] In S2011, the intrinsic shape signatures (ISS) algorithm is used to extract 3D point cloud data from real scene images. The technology of extracting 3D point cloud data from images is already quite mature, and this part is not the focus of this application and will not be described in detail.

[0060] S2012 detects whether the real scene image contains the target plane based on the three-dimensional point cloud data. If so, execute S2013; otherwise, execute S2014.

[0061] At present, the commonly used algorithms for detecting planes from three-dimensional point cloud data mainly include: Random Sample Consensus (RANSAC) algorithm, Region Growing algorithm and Grid-based Region Growing algorithm.

[0062] Among them, the RANSAC algorithm, as the most classic algorithm, can calculate the mathematical model parameters based on a set of sample data sets containing abnormal data, thereby obtaining valid sample data. The basic assumption of the RANSAC algorithm is that the sample data set contains correct data (data that can be described by the mathematical model, also called inliers), and also contains abnormal data (data that deviates far from the normal range and cannot adapt to the data model, also called outliers), that is, the sample data set contains noise. These abnormal data may be caused by incorrect measurements, incorrect assumptions, incorrect calculations, etc. At the same time, the RANSAC algorithm also assumes that, given a set of correct data, there is a method to calculate the model parameters that conform to these data.

[0063] In the process of fitting a plane from 3D point cloud data, small concave and convex points on the plane (such as small potholes on the ground) can be regarded as valid data, which has a certain offset to the fitted plane, while concave and convex points with too large offset (such as obstacles on the ground, deep pits on the ground) are invalid data. Therefore, the RANSAC algorithm can be used to detect whether the target plane is included in the real scene image based on the 3D point cloud data. Compared with the least squares fitting, the RANSAC algorithm aims to obtain the fitting result based on the majority of valid data, so the plane detection effect is better.

[0064] In a specific implementation, multiple planes are randomly fitted according to the three-dimensional point cloud data, and from the multiple planes, the plane containing the least invalid data is selected as the target plane, or the plane containing the most valid data is selected as the target plane, wherein the invalid data is a point in the three-dimensional point cloud data whose distance to the corresponding plane exceeds a preset distance threshold.

[0065] S2013: Establish a three-dimensional space coordinate system with the target plane as the ground.

[0066] In S2013, after the target plane is detected, the position of the target plane is used as the ground position to establish a three-dimensional space coordinate system corresponding to the real scene.

[0067] S2014: Re-collect the real scene image and return to S2011.

[0068] After establishing the three-dimensional space coordinate system, the real objects in the environment are detected using the collected real scene images.

[0069] S202: Determine whether the real scene contains a real object based on the real scene image. If not, execute S203; if yes, execute S204.

[0070] The detection of real objects is similar to ground detection, and requires plane fitting of real objects. For the specific process, see Figure 3, which will not be repeated here.

[0071] S203: placing the virtual character at a preset position in the three-dimensional space coordinate system, and driving the virtual character to move according to preset driving data.

[0072] When the real scene does not contain real objects, the virtual character does not need to interact with real objects. The virtual character can be directly placed at a preset position in the three-dimensional space coordinate system, and preset driving data (including but not limited to voice data and bone data) can be obtained from the cloud, and the preset driving data can be used to drive the movement of the virtual character.

[0073] In a remote three-dimensional communication scenario, when an AR device communicates remotely with other AR devices, the AR device can send a prompt message to the other AR devices that there are no objects in the real scene where it is located. After receiving the prompt message, the other AR devices can determine the movable range of the virtual character corresponding to the local user within the three-dimensional reconstruction area, and synchronize it to the AR device so that the corresponding virtual character can move within the movable range of the three-dimensional space coordinate system.

[0074] S204: Extracting semantic information, distance information, and scale information of real objects according to the real scene image.

[0075] When the real scene contains real objects, the semantic information, distance information and scale information of the real objects are extracted from the real scene image to obtain the specific information of the real objects in the real scene, so that the virtual characters can better adapt to the real scene.

[0076] In the embodiments of the present application, the semantic information of the real object may be the category of the real object. Understanding objects in real scenes based on visual images is not only about detecting and identifying individual objects, but also about being able to use natural language to describe the category of the object expressed by the real scene image. This is because real objects of different categories have different interactive relationships, which will cause the virtual characters to exhibit different behaviors in the real scene image.

[0077] In S204, the semantic information extraction algorithm includes but is not limited to Convolutional Neural Network (CNN), Faster Region Convolutional Neural Network (Faster R-CNN), YOLO network and Single Shot MultiBox Detector (SSD) network. Among them, Faster R-CNN generates target candidate regions through the Region Proposal Network (RPN), which has good detection efficiency. The process of object detection and category recognition using the above networks is already quite mature, so the detailed process of semantic information extraction is no longer expanded.

[0078] In an embodiment of the present application, when a real object is detected, the distance information includes the distance from the real object to the AR device (i.e., the depth information of the object); when multiple real objects are detected, the distance information includes the distance from each real object to the AR device, as well as the distance between the real objects.

[0079] In S204, the distance information of the real object can be extracted by using triangulation method. Figure 4 As shown in the figure, the triangle measurement principle diagram provided in the embodiment of the present application is used to calculate the coordinates X of the object in the three-dimensional space coordinate system through the pixel coordinates x1 and x2 of the corresponding matching points in the two frames of images and the parameters of the multi-eye camera (including the rotation matrix R and the translation vector t). The calculation formula is as follows:

[0080] x1=P1X, x2=P2X Formula 1

[0081] Among them, P1 and P2 are 4*4 matrices composed of the rotation matrix and translation vector between the two cameras.

[0082] In an embodiment of the present application, the parameters of the multi-eye camera on the AR device can be obtained in advance through calibration and can be used as known quantities in S204. The three-dimensional coordinates of the real object in the three-dimensional space coordinate system can be determined through the matching results of the feature points in the real scene image captured by the multi-eye camera of the AR device, so as to determine the distance from each real object to the AR device and the distance between the real objects based on each three-dimensional coordinate.

[0083] S205: Acquire target driving data according to the semantic information of the real object.

[0084] In an embodiment of the present application, each real object is associated with one or more virtual characters, and each voice information corresponds to at least one set of target driving data, as shown in Table 1.

[0085] Table 1. Correspondence between real objects and virtual characters, as well as voice information and target driving data

[0086]

[0087] Among them, the target drive data can be stored in a cloud server or locally on the AR device.

[0088] Taking an example of associating a real object with a virtual character and a voice information with a set of target drive data, assuming that the semantic information of the real object recognized through the real scene image is "TV", the target drive data for enabling the virtual character A to "turn on the TV switch" is obtained based on "TV".

[0089] Taking an example in which a real object is associated with two virtual characters and one voice information corresponds to two sets of target drive data, assuming that the semantic information of the real object recognized through the real scene image is "sofa", then according to "sofa", target drive data 1 for making virtual character A "sit down" is obtained, and target drive data 2 for making virtual character B "sit down and lean down" is obtained.

[0090] S206: placing the virtual character in a three-dimensional space coordinate system according to the distance information, and adjusting the size of the virtual character according to the scale information, so that the virtual character is compatible with the real object.

[0091] For example, assuming that the kitchen scene contains a real object, and the real object identified through the real scene image is a "gas stove", the position of the "gas stove" in the three-dimensional space coordinate system is determined according to the distance from the "gas stove" to the AR device, and the virtual character is placed according to the position of the "gas stove". Furthermore, the size of the virtual character is adjusted according to the scale information of the "gas stove" to make the size ratio of the virtual character and the "gas stove" match, thereby improving the authenticity of the fusion of virtual and real.

[0092] For another example, assuming that an indoor scene contains two real objects, and the real objects identified through the real scene image are "sofa" and "coffee table" respectively, then the positions of the "sofa" and "coffee table" in the three-dimensional space coordinate system are determined according to the distance from the "sofa" to the AR device, the distance from the "coffee table" to the AR device, and the distance between the "sofa" and the "coffee table", and a virtual character is placed according to the positions of the "sofa" and the "coffee table". Furthermore, the size of the virtual character is adjusted according to the scale information of the "sofa" and the "coffee table" to make the size ratio of the virtual character match the "sofa" and the "coffee table", thereby improving the authenticity of the fusion of virtual and real.

[0093] S207: Using the target driving data to drive the virtual character to produce a behavior corresponding to the real object.

[0094] The method for driving virtual characters based on visual semantics provided in the embodiments of the present application can be applied to various AR scenarios in which real scenes and virtual characters are superimposed and displayed, such as AR virtual tours, product introductions, indoor and outdoor navigation, museum introductions, home introductions and the like. The semantic information of objects in the real scene is perceived by the AR device, and the adaptive interaction process with the virtual character is based on the semantic information, so that the virtual character can make corresponding actions according to the real object, rather than just displaying the virtual character. This can reduce some testing and matching losses in the fusion of virtual and real, reduce the sense of disharmony of the combination of virtual characters and real scenes to a certain extent, improve the authenticity of the fusion of virtual and real, and enhance the immersion of virtual character applications and AR experience.

[0095] For example, assuming that the semantic information of the real object recognized through the real scene image is "television", and the acquired target driving data is "turn on the TV switch", the virtual character will perform the action of pressing the TV switch to turn on the TV.

[0096] For another example, assuming that the semantic information of the real object identified through the real scene image is "dangerous goods", and the acquired target driving data is "safety prompt action", the virtual character will generate a prompt action to protect its own safety against the dangerous goods.

[0097] For another example, assuming that the real object identified through the real scene image is a "gas stove", and the target driving data obtained is "cooking", the virtual character will perform the behavior of cooking next to the gas stove.

[0098] For another example, assuming that the real objects identified through the real scene image are "sofas", and the acquired target data is "sit down", the virtual character will perform the behavior of sitting on the sofa.

[0099] It should be noted that there is no restrictive requirement for the generation method of the virtual character in the embodiment of the present application. It can be derived from a simulated character model constructed by images captured by an RGB or RGBD camera, or from a character model bound by manual modeling by an artist, or it can be an open source parametric human body model from a scientific research institution (such as the SMPL model or the SMPL-X model, etc.).

[0100] Considering that each object in a specific scene has its own semantic information, for example, a home scene may have scenes such as a living room, kitchen, and bedroom, and objects in the living room scene include various home appliances (such as TVs and air conditioners) and furniture (such as refrigerators and coffee tables). In this way, in some complex real scenes, multiple types of objects may be detected and identified, and each type of object has different semantics and corresponding drive data. Therefore, the semantic information can be sorted to sequentially drive the virtual character to produce corresponding behaviors.

[0101] See also Figure 5, is a flow chart of another method for driving a virtual character based on visual semantics provided in an embodiment of the present application. The process is executed by an AR device and mainly includes the following steps:

[0102] S501: Acquire a real scene image, and establish a three-dimensional space coordinate system according to the real scene image.

[0103] The specific description of this step refers to the above embodiment and will not be repeated here.

[0104] S502: Determine whether the real scene contains at least one real object according to the real scene image, if not, execute S503, if yes, execute S504.

[0105] In specific implementation, three-dimensional point cloud data of objects are extracted from real scene images, and a RANSAC algorithm is used to perform plane fitting on the three-dimensional point cloud data. By determining the number of inner points or outer points corresponding to each plane, at least one real object contained in the real scene is detected. When no real object is detected, S503 is executed, and when at least one real object is detected, S504 is executed.

[0106] S503: placing the virtual character at a preset position in the three-dimensional space coordinate system, and driving the virtual character to move according to preset driving data.

[0107] When the real scene does not contain real objects, the virtual character does not need to interact with the real objects. The virtual character can be directly placed at a preset position in the three-dimensional space coordinate system, and the obtained preset driving data can be used to drive the virtual character to move within the movable range of the three-dimensional space coordinate system.

[0108] S504: Extracting semantic information, distance information and scale information of each real object according to the real scene image.

[0109] The specific description of this step refers to the above embodiment and will not be repeated here.

[0110] S505: Sort the importance of each real object according to the semantic information of each real object.

[0111] A real scene may contain multiple real objects, each of which has different semantic information and can drive different virtual characters to produce corresponding behaviors. In order to reduce the clutter of the display screen, the virtual characters corresponding to the most important real objects can be driven first. Therefore, the importance of each real object can be sorted according to its semantic information.

[0112] For example, a living room scene includes two real objects, a sofa and a TV. According to human living habits, people usually sit on the sofa first and then turn on the TV. Therefore, the sofa is more important than the TV.

[0113] S506: Obtain target driving data corresponding to the semantic information of each real object in order from high to low importance.

[0114] In S506 , according to the importance of each real object, target driving data corresponding to semantic information with higher importance is obtained in a limited manner, and each obtained target driving data is stored in a queue.

[0115] For example, assuming that the living room scene contains two real objects, a sofa and a TV, and the sofa is more important than the TV, the target drive data corresponding to the sofa is obtained first, and then the target drive data corresponding to the TV is obtained.

[0116] S507: placing each virtual character in a three-dimensional space coordinate system according to the distance information, and adjusting the size of each virtual character according to the scale information, so that each virtual character is compatible with the real object.

[0117] The specific description of this step refers to the above embodiment and will not be repeated here.

[0118] S508: According to the acquisition order of the target driving data, the corresponding virtual characters are driven in sequence to produce corresponding behaviors corresponding to the real objects.

[0119] For example, assuming that the living room scene contains two real objects, a sofa and a TV, and the target drive data corresponding to the sofa is obtained before the target drive data corresponding to the TV, the virtual character A corresponding to the sofa is first driven to sit on the sofa, and then the virtual character B is driven to turn on the TV.

[0120] In an embodiment of the present application, the storage format of the target drive data can be a bvh data format, wherein bvh is a universal human feature animation file format that can be supported by various animation production software, and the target drive data includes the character's bone data and the rotation data of the limb joints.

[0121] In the method for driving a virtual character based on visual semantics provided in an embodiment of the present application, the semantic information, distance information and scale information of the real object are extracted by using the real scene image collected by the multi-eye camera of the AR device, the virtual character is placed in the three-dimensional space coordinate system according to the distance information, and the size of the virtual character is adjusted according to the scale information. By sorting the semantic information of each real object, the importance of each real object can be determined, so as to orderly obtain the target driving data corresponding to the real objects with higher importance to drive the virtual character to produce corresponding behavior, realize automatic interaction between the virtual character and the real object, reduce the sense of disharmony of the combination of the virtual character and the real scene to a certain extent, improve the authenticity of the fusion of virtual and real, and thus enhance the user's AR experience.

[0122] In some embodiments, when placing each virtual character in a three-dimensional space coordinate system, collision detection can be performed between models to prevent occlusion between models during the interaction process. For the specific process, see Figure 6 , mainly includes the following steps:

[0123] S601: Determine the position of each virtual character's foot in the three-dimensional space coordinate system in sequence according to the distance information.

[0124] In S601, the plane where the feet of the virtual character are located can be used as the ground of the three-dimensional space coordinate system, and the position of each virtual character's foot in the three-dimensional space coordinate system can be determined in sequence according to the order in the model list, the distance from each real object to the AR device and the distance between the real objects.

[0125] S602: Move each virtual character according to a preset threshold value to determine the activity range of the multiple virtual characters in the three-dimensional space coordinate system.

[0126] In an embodiment of the present application, a three-dimensional space coordinate system is established based on the collected real scene images, and the position of the real object in the three-dimensional space coordinate system is determined with the ground where the virtual character's feet are located, and each virtual character is moved in the three-dimensional space coordinate system according to a preset threshold. The movement is stopped when a collision is detected with a real object or other virtual characters, thereby determining the activity range of multiple virtual characters in the three-dimensional space coordinate system.

[0127] Among them, the collision detection process can use AABB bounding boxes or OBB bounding boxes to transform the object intersection problem in three-dimensional space into two-dimensional or one-dimensional space for processing, and improve the efficiency of processing problems by reducing the dimension. The embodiment of the present application takes the transformation of the overlap test problem of the bounding boxes of virtual characters and real objects in three-dimensional space into one-dimensional space as an example. If two bounding boxes overlap, their projected segments on the three coordinate axes overlap. This is because the AABB bounding box is a convex polyhedron, so if the projected segments of the two convex polyhedrons on the three coordinate axes overlap, then they must intersect.

[0128] From the definition of AABB bounding box, we know that the points with minimum and maximum coordinate values ​​on the AABB bounding box must be vertices, and the projection points of these vertices on the three coordinate axes can form the minimum and maximum projection line segments on the corresponding coordinate axes. It is assumed that the vertex with the minimum coordinate value is called the minimum vertex, and the vertex with the maximum coordinate value is called the maximum vertex. In this way, to determine whether two bounding boxes intersect, we only need to compare the three-dimensional coordinate values ​​of the four points of the minimum vertex and the maximum vertex of the two bounding boxes.

[0129] S603: Divide the activity range equally according to the number of virtual characters to obtain an activity sub-range of each virtual character.

[0130] In S603, the activity range is evenly divided according to the number of people to obtain the activity sub-range of each virtual character.

[0131] See also Fig. 7A , which is a rendering of a virtual character driven based on visual semantics provided by an embodiment of the present application in a bedroom scene, such as Fig. 7A As shown, the bedroom scene contains objects such as chairs and beds. The target driving data 1 obtained based on the "bed" is used to drive two virtual characters to sit down, the target driving data obtained based on the "chair" is used to drive one virtual character to lean on the back of the chair, the target driving data 2 obtained based on the "bed" is used to drive another virtual character to walk towards the bed, and the target driving data 2 obtained based on the "chair" is used to drive another virtual character to stand next to the chair.

[0132] See also Figure 7B , which is a rendering of a virtual character driven based on visual semantics provided by an embodiment of the present application in a living room scene, such as Figure 7B As shown, the living room scene includes objects such as a sofa, a wall, a coffee table, and a TV. Three sets of target driving data obtained based on the "sofa" are used to drive three virtual characters to sit on the sofa respectively, and the target driving data obtained based on the "wall" is used to drive a virtual character to lean against the wall, and another virtual character to walk between the TV and the coffee table.

[0133] like Fig. 7A and Figure 7B As shown, during the process of automatic interaction between the virtual characters and the soil in the real scene, there is no mutual occlusion between the virtual characters, and between the virtual characters and the real objects.

[0134] Based on the same technical concept, an embodiment of the present application provides an AR device, which can implement the method steps of driving a virtual character based on visual semantics in the above embodiment and achieve the same technical effect.

[0135] See also Figure 8 , the AR device includes a multi-eye camera 801, a processor 802, a memory 803 and a display screen 804, wherein the display screen 804, the memory 803, the multi-eye camera 801 and the processor 802 are connected via a bus 805:

[0136] The multi-eye camera 801 is used to collect real scene images;

[0137] The memory 803 includes a data storage unit and a program storage unit. The program storage unit stores a computer program. The processor 802 performs the following operations according to the computer program:

[0138] Acquire a real scene image, display it on the display screen 804, and establish a three-dimensional space coordinate system according to the real scene image;

[0139] When a real object is detected according to the real scene image, semantic information, distance information and scale information of the real object are extracted;

[0140] Acquire target driving data according to the semantic information of the real object, and store it in the data storage unit;

[0141] placing the virtual character in the three-dimensional space coordinate system according to the distance information, and adjusting the size of the virtual character according to the scale information so that the virtual character fits the real object;

[0142] The target driving data is used to drive the virtual character to generate a behavior corresponding to the real object, and the driven virtual character is superimposed and displayed on the display screen 804 .

[0143] Optionally, when multiple real objects are detected, the processor 802 obtains target drive data according to semantic information of the real objects, and the specific operation is:

[0144] Sort the importance of multiple real objects according to their semantic information;

[0145] The target driving data corresponding to the semantic information of each real object is obtained in sequence from high to low importance.

[0146] Optionally, the processor 802 drives the virtual character to generate a behavior corresponding to the real object by using the target driving data, and the specific operation is:

[0147] According to the acquisition order of the target driving data, the corresponding virtual characters are driven in sequence to produce corresponding behaviors corresponding to the real objects.

[0148] Optionally, when there are multiple virtual characters, when the processor 802 places the virtual characters in the three-dimensional space coordinate system according to the distance information, it further executes:

[0149] Determining the position of each virtual character's foot in the three-dimensional space coordinate system in sequence according to the distance information;

[0150] Move each virtual character according to a preset threshold value to determine the activity range of the multiple virtual characters in the three-dimensional space coordinate system;

[0151] The activity range is equally divided according to the number of the virtual characters to obtain an activity sub-range for each virtual character.

[0152] Optionally, when a real object is detected, the distance information includes the distance from the real object to the AR device;

[0153] When multiple real objects are detected, the distance information includes the distance from each real object to the AR device, and the distance between the real objects.

[0154] Optionally, each real object is associated with one or more virtual characters, and each semantic information corresponds to at least one set of target driving data.

[0155] It should be noted that Figure 8 This is just an example, and the necessary hardware for an AR device to execute the method steps of driving a virtual character based on visual semantics provided in the embodiments of the present application is given. The AR device not shown also includes common hardware of a head-mounted display (HMD), such as left and right eyeglasses, speakers, microphones, sensors (such as a relationship measurement unit IMU), and a communication interface connected to devices such as handles.

[0156] Embodiments of the present application Figure 8 The processor involved may be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0157] See also Fig. 9 ,for Figure 8 The functional structure diagram of the AR device shown in FIG. 1 includes an image acquisition module 901, an information extraction module 902, a data acquisition module 903, and a display driver module 904, wherein:

[0158] An image acquisition module 901 is used to acquire a real scene image and establish a three-dimensional space coordinate system according to the real scene image;

[0159] An information extraction module 902 is used to extract semantic information, distance information and scale information of a real object when a real object is detected according to the real scene image;

[0160] A data acquisition module 903 is used to acquire target driving data according to the semantic information of the real object and store it in the data storage unit;

[0161] The display driving module 904 is used to place the virtual character in the three-dimensional space coordinate system according to the distance information, and adjust the size of the virtual character according to the scale information so that the virtual character is compatible with the real object; and use the target driving data to drive the virtual character to generate corresponding behavior of the real object.

[0162] The above functional modules cooperate with each other to realize the following method steps of driving a virtual character based on visual semantics, and can achieve the same technical effect. The specific implementation of each functional module can be found in the above embodiment, which will not be repeated here.

[0163] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the method of the aforementioned embodiment.

[0164] The embodiments of the present application also provide a computer program product for storing a computer program, wherein the computer program is used to execute the method of the aforementioned embodiment.

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0166] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0167] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0168] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0169] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for driving a virtual character based on visual semantics, characterized in that: Applied to an AR device, the method includes: Acquire a real scene image, and establish a three-dimensional space coordinate system according to the real scene image; When a real object is detected according to the real scene image, the semantic information, distance information and scale information of the real object are extracted, wherein each semantic information corresponds to at least one set of target drive data; the distance information includes the distance from the real object to the AR device and / or the distance between the real objects, and the distance information is obtained in the following manner: the three-dimensional coordinates of the real object in the three-dimensional space coordinate system are determined according to the matching results of the feature points in the real scene image, and the distance from each real object to the AR device and the distance between the real objects are determined according to each three-dimensional coordinate; when multiple real objects are detected, the semantic information of the multiple real objects is sorted according to the importance; and the target drive data corresponding to the semantic information of each real object is obtained in order from high to low importance; wherein the target drive data includes the skeleton data of the user corresponding to the virtual character and the rotation data of the limb joints; placing the virtual character in the three-dimensional space coordinate system according to the distance information, and adjusting the size of the virtual character according to the scale information so that the virtual character fits the real object; The target driving data is used to drive the virtual character to produce a behavior corresponding to the real object.

2. The method according to claim 1, characterized in that The using the target driving data to drive the virtual character to generate the behavior corresponding to the real object includes: According to the acquisition order of the target driving data, the corresponding virtual characters are driven in sequence to produce corresponding behaviors corresponding to the real objects.

3. The method according to claim 1, characterized in that When there are multiple virtual characters, when placing the virtual characters in the three-dimensional space coordinate system according to the distance information, the method further includes: Determining the position of each virtual character's foot in the three-dimensional space coordinate system in sequence according to the distance information; Move each virtual character according to a preset threshold value to determine the activity range of the multiple virtual characters in the three-dimensional space coordinate system; The activity range is equally divided according to the number of the virtual characters to obtain an activity sub-range for each virtual character.

4. The method according to any one of claims 1 to 3, characterized in that When a real object is detected, the distance information includes the distance from the real object to the AR device; When multiple real objects are detected, the distance information includes the distance from each real object to the AR device, and the distance between the real objects.

5. The method according to any one of claims 1 to 3, characterized in that Each real object is associated with one or more virtual characters, and each semantic information corresponds to at least one set of target driving data.

6. An AR device, characterized in that: It includes a multi-eye camera, a processor, a memory and a display screen, wherein the display screen, the memory, the multi-eye camera and the processor are connected via a bus: The multi-eye camera is used to collect real scene images; The memory includes a data storage unit and a program storage unit, the program storage unit stores a computer program, and the processor performs the following operations according to the computer program: Acquire a real scene image, display it on the display screen, and establish a three-dimensional space coordinate system based on the real scene image; When a real object is detected according to the real scene image, semantic information, distance information and scale information of the real object are extracted, wherein each semantic information corresponds to at least one set of target drive data; the distance information includes the distance from the real object to the AR device and / or the distance between the real objects, and the distance information is obtained in the following manner: determining the three-dimensional coordinates of the real object in the three-dimensional space coordinate system through the matching results of the feature points in the real scene image, and determining the distance from each real object to the AR device and the distance between the real objects according to the three-dimensional coordinates; When multiple real objects are detected, the semantic information of the multiple real objects is sorted according to their importance; the target drive data corresponding to the semantic information of each real object is obtained in order from high to low importance, and stored in the data storage unit; wherein the target drive data includes the skeleton data and the rotation data of the limb joints of the user corresponding to the virtual character; placing the virtual character in the three-dimensional space coordinate system according to the distance information, and adjusting the size of the virtual character according to the scale information so that the virtual character fits the real object; The target driving data is used to drive the virtual character to generate a behavior corresponding to the real object, and the driven virtual character is displayed by overlaying on the display screen.

7. The AR device according to claim 6, characterized in that: The processor drives the virtual character to generate a behavior corresponding to the real object by using the target driving data, and the specific operation is: According to the acquisition order of the target driving data, the corresponding virtual characters are driven in sequence to produce corresponding behaviors corresponding to the real objects.

8. The AR device according to any one of claims 6 to 7, characterized in that: Each real object is associated with one or more virtual characters, and each semantic information corresponds to at least one set of target driving data.

Images