Synchronous mixed reality method with immersion and interaction efficiency

Abstract

The invention provides a synchronous mixed reality method with immersion and interaction efficiency, and relates to the technical field of virtual-real fusion and deep learning, and the method comprises the steps: firstly, employing a deep learning instance segmentation algorithm to extract the information of a physical instance in real time, carrying out the virtual-real fusion, and pre-judging a user interaction purpose based on a bounding box intersection-to-union algorithm, according to an interaction purpose of a user, visual information of an interaction target is directly expressed by adopting a pixel-by-pixel drawing method, visual information of a non-interaction object is indirectly expressed by adopting a characterization icon drawing method, and finally selective expression of physical space target information in a virtual environment is realized. According to the method, the target interaction precision is improved on the basis of ensuring the environment immersion, the environment performance and the system interaction efficiency can be considered, the rendering freedom degree of the virtual-real fusion scene is expanded, and intelligent fusion of the virtual-real space is realized.

Description

Technical Field

[0001] The present invention relates to the field of virtual-reality fusion and deep learning technology, and more specifically, to a synchronous mixed reality method with both immersion and interactive performance. Background Art

[0002] Synchronized Mixed Reality (SMR) refers to the use of immersive head-mounted display devices to build a "synchronized space", match virtual and real objects into "synchronized interactive objects" with common attributes through video perspective, and express the synchronized interactive objects in real time in the synchronized space to achieve "object fusion" and "environment synchronization".

[0003] "Synchronous space" refers to a unified space that has both virtual space characteristics and physical space dimensions. This space is built based on immersive display devices and uses video perspective to overlay information. According to different information overlay ratios, it can realize intelligent changes from pure virtual environment to pure physical environment. "Synchronous interactive object" refers to an interactive object existing in the synchronous space, which is composed of a physical entity in the physical space and a virtual agent in the virtual space. The physical entity has the functional attributes of the object itself (such as the keyboard has input function), and the virtual agent has both the visual presentation attributes of the physical object (visual presentation of the keyboard) and the virtual information overlay attributes (graphic drawing and functional redefinition of keyboard keys). The combination of physical entity and virtual agent makes the synchronous interactive object have both functional penetration characteristics and information overlay and functional expansion characteristics, so as to expand the functions of the virtual-real fusion system and reduce the difficulty of building complex systems. "Object fusion" refers to matching the physical entity and the virtual agent with a certain spatial conversion relationship after extracting information from the physical target to achieve the time and space synchronization of the two. "Environment synchronization" refers to the real-time synchronization of the physical environment and the virtual environment through the conversion of the virtual space coordinate system and the physical space coordinate system, and the spatial synchronization of the interactive object characteristics (position, posture, deformation, temperature and other information) through the drawing or superposition of visual cues.

[0004] Traditional virtual-reality fusion technologies include augmented virtual technologies that use physical objects as fusion objects, such as Figure 1 As shown; and augmented reality technology with virtual objects as fusion objects, such as Figure 2 Compared with traditional technologies, synchronous mixed reality technology uses space as the object for virtual-real fusion, which not only includes the real-time fusion of virtual objects and physical objects, but also includes the feature fusion of physical space and virtual space, and is committed to achieving Figure 3The "intelligent synchronization", "spatial synchronization" and "temporal synchronization" between the user, physical space and virtual space are shown. "Intelligent synchronization" emphasizes the two-way synchronization mechanism between physical objects and virtual objects; unlike the existing fusion method with unchanged virtual-real ratio, synchronous mixed reality technology builds a new immersive environment with equal virtual-real relationship in a unified space according to user needs, and can intelligently adjust the fusion ratio of virtual information and physical information, thereby realizing intelligent virtual-real fusion from pure physical space to pure virtual space. "Spatial synchronization" emphasizes the consistency of virtual space and physical space in terrain and object shape structure, that is, in the new immersive environment, the entity objects in the physical space have a virtual agent, and the two maintain spatial consistency so that what the user sees is what he gets. "Temporal synchronization" emphasizes the system's ability to perceive and manipulate dynamic changes in the environment, requiring the system to monitor the changes in the visual information and posture information of physical objects in real time and synchronize them to the corresponding virtual objects.

[0005] Synchronous mixed reality technology can extract and fuse information in any environment, meet different system functions, reduce the cost of virtual-real fusion while improving the degree of freedom of virtual-real fusion, and has high application value. However, most of the current synchronous mixed reality technology remains in the theoretical stage. Since it cannot achieve both environmental immersion and system interaction efficiency when drawing virtual-real fusion scenes, its development has encountered bottlenecks and is difficult to be widely used. Summary of the invention

[0006] In order to overcome the defect of the above-mentioned prior art that it is unable to have both environmental immersion and system interaction efficiency when drawing virtual-reality fusion scenes, the present invention provides a synchronous mixed reality method with both immersion and interaction efficiency. The selective expression of target information is achieved through the intersection-over-union algorithm, and the target interaction accuracy is improved on the basis of ensuring environmental immersion, the rendering freedom of virtual-reality fusion scenes is expanded, and the intelligent fusion of virtual and real spaces is achieved.

[0007] In order to solve the above technical problems, the technical solution of the present invention is as follows: A synchronous mixed reality method with both immersion and interactive performance includes the following steps: S1: Use a camera to acquire a physical scene image, and use a pre-trained instance segmentation model to extract visual information, pose information, and segmentation bounding boxes of several physical instances from the physical scene image; The physical instances include: active movable objects and passive movable objects; S2: Load the virtual environment, draw direct clues for the visual information of the active movable object, draw characteristic indirect clues for the visual information of the passive movable object, and obtain the virtual object corresponding to each physical instance; S3: Obtain the position and posture information of each virtual object in the virtual environment, and map the virtual and real coordinates with the position and posture information of the corresponding physical instance to complete the rendering of the virtual and real fusion scene; S4: Based on the rendered virtual-reality fusion scene, calculate the intersection-over-union ratio between the segmentation bounding box of the active movable object and the segmentation bounding box of each passive movable object, predict the user's interaction target according to the calculation result, and classify all passive movable objects into targets to be interacted and non-interaction targets; S5: redrawing the visual information of the target to be interacted with by direct clue drawing, obtaining the secondary drawn virtual object corresponding to the target to be interacted with and performing visual expression; for the virtual objects corresponding to the active movable objects and non-interactive targets, performing visual expression directly; S6: Repeat steps S4 to S6 to complete the synchronous mixed reality of selectively expressing the target information.

[0008] Preferably, in step S1, the pre-trained instance segmentation model is specifically: a YOLACT instance segmentation model; The YOLACT instance segmentation model includes: a target detection network RetinaNet, and a mask coefficient prediction network and a fully convolutional neural network FCN set in parallel; the output of the target detection network RetinaNet is connected to the input of the mask coefficient prediction network and the fully convolutional neural network FCN respectively; The output of the mask coefficient prediction network is processed by the non-maximum suppression algorithm NMS, and then combined with the output of the fully convolutional neural network FCN to obtain the output of the YOLACT instance segmentation model.

[0009] Preferably, in the YOLACT instance segmentation model, the target detection network RetinaNet includes a RestNet-101 residual subnetwork and a feature pyramid subnetwork FPN connected in sequence; The mask coefficient prediction network includes a classification subnetwork, a frame regression subnetwork and a mask coefficient prediction subnetwork arranged in parallel; The fully convolutional neural network FCN is specifically a Protonet prototype mask generation subnetwork; The non-maximum suppression algorithm NMS is specifically a Fast NMS algorithm.

[0010] Preferably, the output of the mask coefficient prediction network is processed by a non-maximum suppression algorithm NMS to obtain The coefficient matrix of , where n is the number of physical instances calculated by the non-maximum suppression algorithm NMS, and k is the mask coefficient of each physical instance; The output of the fully convolutional neural network FCN is The prototype mask matrix , where h and w are the first and second sizes of the prototype mask, respectively. The number of prototype masks for each physical instance; The coefficient matrix and the prototype mask matrix Perform a combined operation according to the following formula to obtain the output of the YOLACT instance segmentation model :

[0011] in, is the Sigmoid function.

[0012] Preferably, the loss function of the YOLACT instance segmentation model in the pre-training process includes: classification loss function , box regression loss function And the mask coefficient prediction loss function ; The classification loss function Specifically, it is the Softmax cross entropy loss function; The box regression loss function Includes: Confidence loss and position loss , the confidence loss Specifically, it is the Softmax cross entropy loss function, the position loss Specifically, it is the Smooth L1 loss function; The mask coefficient prediction loss function Specifically, it is a binary cross entropy loss function.

[0013] Preferably, the step S1 further comprises: setting a category hash table, wherein the category hash table stores several categories of physical instances that can be integrated into the virtual environment; Determine whether each extracted physical instance belongs to the physical instance category in the category hash table. If so, directly execute step S2; if not, delete the visual information, posture information and segmentation bounding box of the physical instance and execute step S2.

[0014] Preferably, in step S2 and step S5, direct clue drawing includes: physical instance ontology drawing and background information drawing; The physical instance ontology drawing specifically includes: drawing the outline, texture and color of the physical instance pixel by pixel in the virtual space according to the visual information of the physical instance, and obtaining the corresponding virtual object; The background information drawing is specifically as follows: taking pixels other than the pixels of the physical instance as background pixels, setting the pixel values ​​of the background pixels to 0, and completing the background information drawing.

[0015] Preferably, in step S2, characterizing indirect clue drawing includes: The size of the physical instance and the coordinates of the center point position are obtained according to the visual information of the physical instance, and the visual information of the physical instance is replaced with a preset virtual feature map to obtain a corresponding virtual object; The size and center point position coordinates of the preset virtual feature map are the same as those of the physical instance.

[0016] Preferably, in step S4, the intersection-and-union ratio between the segmentation bounding box A of the active movable object and the segmentation bounding box B of each passive movable object is calculated according to the following formula: :

[0017] in, The intersection of segmentation bounding box A and segmentation bounding box B is x The length of the side on the axis; The intersection of segmentation bounding box A and segmentation bounding box B is y The length of the side on the axis; is the coordinate of the lower left corner vertex of the segmentation bounding box A; is the coordinate of the upper right corner vertex of the segmentation bounding box A; is the coordinate of the lower left corner vertex of the segmentation bounding box B; is the coordinate of the upper right corner vertex of the segmentation bounding box B.

[0018] Preferably, in step S4, predicting the user's interaction target according to the calculation result includes: It is determined whether the intersection-and-union ratio between the segmentation bounding box of the active movable object and the segmentation bounding box of each passive movable object is greater than or equal to a preset threshold. If so, the category of the passive movable object is a target to be interacted with; if less than, the category of the passive movable object is a non-interactive target.

[0019] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The present invention provides a synchronous mixed reality method with both immersion and interactive performance. First, a physical scene image is acquired by a camera, and a pre-trained instance segmentation model is used to extract visual information, posture information and segmentation boundary boxes of several physical instances from the physical scene image; the physical instances include: active movable objects and passive movable objects; a virtual environment is loaded, and direct clues are drawn for the visual information of the active movable objects; and characteristic indirect clues are drawn for the visual information of the passive movable objects; a virtual object corresponding to each physical instance is acquired; the posture information of each virtual object in the virtual environment is acquired, and the posture information of the corresponding physical instance is respectively compared with the posture information of the virtual object in the virtual environment. The information is mapped into virtual and real coordinates to complete the rendering of the virtual and real fusion scene; based on the rendered virtual and real fusion scene, the intersection and union ratio between the segmentation bounding box of the active movable object and the segmentation bounding box of each passive movable object is calculated, and the user's interaction target is predicted according to the calculation result, and all passive movable objects are classified into targets to be interacted and non-interactive targets; the visual information of the target to be interacted is redrawn by direct clue drawing, and the secondary drawn virtual object corresponding to the target to be interacted is obtained and visually expressed; for the virtual objects corresponding to the active movable object and the non-interactive target, the visual expression is directly performed; the above steps are repeated to complete the synchronous mixed reality with selective expression of target information; The present invention adopts a deep learning instance segmentation algorithm to extract the information of physical instances in real time, and predicts the purpose of user interaction based on the bounding box intersection-union algorithm after virtual-real fusion, thereby realizing the selective expression of physical target information in the virtual environment. It improves the target interaction accuracy while ensuring the environmental immersion, and can take into account both environmental performance and system interaction efficiency. At the same time, it expands the rendering freedom of the virtual-real fusion scene and realizes the intelligent fusion of virtual and real space. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of the enhanced virtual technology provided in the background technology.

[0021] Figure 2 This is a schematic diagram of the augmented reality technology provided in the background technology.

[0022] Figure 3 This is a schematic diagram of the synchronous mixed reality technology provided in the background technology.

[0023] Figure 4 This is a flow chart of a synchronous mixed reality method with both immersion and interactive performance provided in Example 1.

[0024] Figure 5 This is an implementation architecture diagram of the synchronous mixed reality technology provided in Example 1.

[0025] Figure 6This is a structural diagram of the YOLACT instance segmentation model provided in Example 2.

[0026] Figure 7 This is a structural diagram of the target detection network RetinaNet provided in Example 2.

[0027] Figure 8 This is a structural diagram of the residual bottleneck block in the RestNet-101 residual subnetwork provided in Example 2.

[0028] Fig. 9 Generate a subnetwork structure diagram for the Protonet prototype mask provided in Example 2.

[0029] Fig.10 This is a diagram of the mask coefficient prediction network structure provided in Example 2.

[0030] Fig.11 This is a schematic diagram of intelligently extracting target information after setting the category hash table provided in Example 2.

[0031] Fig.12 A schematic diagram is drawn for direct clues of the virtual-reality fusion scene provided in Example 2.

[0032] Fig.13 A schematic diagram of the indirect clues for characterizing the virtual-reality fusion scene provided in Example 2 is drawn.

[0033] Fig.14 A schematic diagram is drawn for visualizing clues of different target object information provided in Example 2.

[0034] Fig.15 This is a schematic diagram showing the side length of the intersection of two bounding boxes provided in Example 2.

[0035] Fig.16 This is a schematic diagram of the selective expression of target information based on the predicted interaction purpose provided in Example 2.

[0036] Fig.17 This is a schematic diagram of scene rendering using a synchronous mixed reality hardware system provided in Example 3. DETAILED DESCRIPTION

[0037] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present application; In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0038] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0039] Example 1 like Figure 4 As shown, this embodiment provides a synchronous mixed reality method with both immersion and interactive performance, including the following steps: S1: Use a camera to acquire a physical scene image, and use a pre-trained instance segmentation model to extract visual information, pose information, and segmentation bounding boxes of several physical instances from the physical scene image; The physical instances include: active movable objects and passive movable objects; S2: Load the virtual environment, draw direct clues for the visual information of the active movable object, draw characteristic indirect clues for the visual information of the passive movable object, and obtain the virtual object corresponding to each physical instance; S3: Obtain the position and posture information of each virtual object in the virtual environment, and map the virtual and real coordinates with the position and posture information of the corresponding physical instance to complete the rendering of the virtual and real fusion scene; S4: Based on the rendered virtual-reality fusion scene, calculate the intersection-over-union ratio between the segmentation bounding box of the active movable object and the segmentation bounding box of each passive movable object, predict the user's interaction target according to the calculation result, and classify all passive movable objects into targets to be interacted and non-interaction targets; S5: redrawing the visual information of the target to be interacted with by direct clue drawing, obtaining the secondary drawn virtual object corresponding to the target to be interacted with and performing visual expression; for the virtual objects corresponding to the active movable objects and non-interactive targets, performing visual expression directly; S6: Repeat steps S4 to S6 to complete the synchronous mixed reality of selectively expressing the target information.

[0040] In the specific implementation process, Figure 5 The figure shows the implementation architecture of synchronous mixed reality technology, which mainly includes four key steps: physical target information extraction, target characterization drawing, virtual-reality mapping and target selective spatiotemporal synchronization. The target information extraction process extracts the visual information and posture information of the target in the physical scene through the image segmentation method; the target characterization drawing extracts the feature information of the virtual object and assigns it to the physical target to achieve the visual feature synchronization of the virtual and real objects; the virtual-reality mapping transforms the physical space coordinate system with the virtual space coordinate system to achieve target posture tracking; the target selective spatiotemporal synchronization is to selectively express specific target information in the virtual environment according to the needs of the user; In the physical target information extraction based on instance segmentation, the physical space information is first captured by an RGB camera and a deep learning instance segmentation algorithm is used to calculate the category, bounding box, and mask prototype information of the physical instance, and obtain its visual information and pose information; In order to achieve seamless integration of physical information in the virtual environment, the physical objects in the information extraction space need to be classified before building the virtual-real integration environment; according to the motion characteristics, interaction paradigm and existence status of physical objects, the target objects in the physical space can be divided into active movable objects (AMO), passive movable objects (PMO) and immovable objects (IMO), and the priority of the three types of information decreases in turn; The first level is active movable objects represented by humans, animals, robots, etc. This type of object has subjective initiative, and the movement space and activity trajectory are determined by the moving object itself. It has the characteristics of high freedom of movement, large range of activity, and uncontrolled existence. In addition, this type of object has obvious personalized characteristics and the personalized characteristics have an important impact on interaction. In the virtual-reality fusion scene, the most typical representative of this type of object is the user himself. They are not only the object of information extraction, but also the sender and executor of interactive behavior, and have the highest interaction weight and information presentation level. In addition to users, objects at this level also include environmental participants, robots that can move freely, pets and other physical objects with active movement capabilities. The interaction weight and existence state of this type of object in the physical space are uncontrollable. Therefore, in order to improve the interaction efficiency of the virtual-reality fusion system and reduce environmental collisions, this type of object has the highest information extraction priority when extracting information. The second level is passive movable objects represented by interactive tools, such as keyboards, mobile phones, cups and other objects with specific functions but uncertain motion states. Such objects have controllable motion trajectories and activity ranges, and their appearance and disappearance in physical space are stable. Although they have personalized characteristics, information such as visual texture, color, and geometric shape has little impact on the interaction. Therefore, real-time drawing is not required, but real-time tracking is required. The third level is immovable objects such as walls and tables. The spatial position of such objects is generally fixed and the interaction weight is low, so they do not need to be accurately expressed. However, in order to avoid collisions, the user's movable space needs to be defined based on the third level objects. In this embodiment, the extracted physical instances only retain active movable objects and passive movable objects, and the visual information of the extracted immovable objects is not subsequently drawn; In the process of target characterization drawing, it is necessary to first build a physical target label library according to the target category, then use the feature extraction method to calculate the virtual environment features and assign the virtual feature information to the physical target label to realize the feature presentation of the label library, and finally use the feature label to replace the physical information to realize the unification of the visual features of the physical target and the virtual environment, so as to ensure the immersion of the virtual and real fusion environment; In the virtual-to-real mapping process, the camera coordinate system of the virtual space is converted into the camera coordinate system of the physical space through space transformation to achieve the unification of the position, direction and distance of the virtual space and the physical space; During the selective spatiotemporal synchronization of target information, the bounding box intersection-union algorithm is first used to predict the purpose of user interaction based on the target depth occlusion relationship. Secondly, according to the user interaction purpose, a pixel-by-pixel drawing method is used to directly express the visual information of the interactive target, and a feature-based icon drawing method is used to indirectly express the visual information of the non-interactive object. Finally, the selective expression of physical space target information in the virtual environment is achieved, thereby improving the system interaction efficiency while ensuring the immersiveness of the virtual-reality fusion environment.

[0041] Example 2 This embodiment provides a synchronous mixed reality method with both immersion and interactive performance, including the following steps: S1: Use a camera to acquire a physical scene image, and use a pre-trained instance segmentation model to extract visual information, pose information, and segmentation bounding boxes of several physical instances from the physical scene image; The physical instances include: active movable objects and passive movable objects; S2: Load the virtual environment, draw direct clues for the visual information of the active movable object, draw characteristic indirect clues for the visual information of the passive movable object, and obtain the virtual object corresponding to each physical instance; S3: Obtain the position and posture information of each virtual object in the virtual environment, and map the virtual and real coordinates with the position and posture information of the corresponding physical instance to complete the rendering of the virtual and real fusion scene; S4: Based on the rendered virtual-reality fusion scene, calculate the intersection-over-union ratio between the segmentation bounding box of the active movable object and the segmentation bounding box of each passive movable object, predict the user's interaction target according to the calculation result, and classify all passive movable objects into targets to be interacted and non-interaction targets; S5: redrawing the visual information of the target to be interacted with by direct clue drawing, obtaining the secondary drawn virtual object corresponding to the target to be interacted with and performing visual expression; for the virtual objects corresponding to the active movable objects and non-interactive targets, performing visual expression directly; S6: repeating steps S4 to S6 to complete the synchronous mixed reality of selectively expressing the target information; In step S1, the pre-trained instance segmentation model is specifically: a YOLACT instance segmentation model; The YOLACT instance segmentation model includes: a target detection network RetinaNet, and a mask coefficient prediction network and a fully convolutional neural network FCN set in parallel; the output of the target detection network RetinaNet is connected to the input of the mask coefficient prediction network and the fully convolutional neural network FCN respectively; The output of the mask coefficient prediction network is processed by the non-maximum suppression algorithm NMS, and then combined with the output of the fully convolutional neural network FCN to obtain the output of the YOLACT instance segmentation model; In the YOLACT instance segmentation model, the target detection network RetinaNet includes a RestNet-101 residual subnetwork and a feature pyramid subnetwork FPN connected in sequence; The mask coefficient prediction network includes a classification subnetwork, a frame regression subnetwork and a mask coefficient prediction subnetwork arranged in parallel; The fully convolutional neural network FCN is specifically a Protonet prototype mask generation subnetwork; The non-maximum suppression algorithm NMS is specifically a Fast NMS algorithm; The output of the mask coefficient prediction network is processed by the non-maximum suppression algorithm NMS to obtain The coefficient matrix of , where n is the number of physical instances calculated by the non-maximum suppression algorithm NMS, and k is the mask coefficient of each physical instance; The output of the fully convolutional neural network FCN is The prototype mask matrix , where h and w are the first and second sizes of the prototype mask, respectively. The number of prototype masks for each physical instance; The coefficient matrix and the prototype mask matrix Perform a combined operation according to the following formula to obtain the output of the YOLACT instance segmentation model :

[0042] in, is the Sigmoid function; The loss functions of the YOLACT instance segmentation model in the pre-training process include: classification loss function , box regression loss function And the mask coefficient prediction loss function ; The classification loss function Specifically, it is the Softmax cross entropy loss function; The box regression loss function Includes: Confidence loss and position loss , the confidence loss Specifically, it is the Softmax cross entropy loss function, the position loss Specifically, it is the Smooth L1 loss function; The mask coefficient prediction loss function Specifically, it is a binary cross entropy loss function; The step S1 further includes: setting a category hash table, wherein the category hash table stores several categories of physical instances that can be integrated into the virtual environment; Determine whether each physical instance extracted belongs to the physical instance category in the category hash table. If so, directly execute step S2; if not, delete the visual information, pose information and segmentation bounding box of the physical instance and execute step S2; In the step S2 and step S5, direct clue drawing includes: physical instance body drawing and background information drawing; The physical instance ontology drawing specifically includes: drawing the outline, texture and color of the physical instance pixel by pixel in the virtual space according to the visual information of the physical instance, and obtaining the corresponding virtual object; The background information drawing is specifically as follows: taking pixels other than the pixels of the physical instance as background pixels, setting the pixel values ​​of the background pixels to 0, and completing the background information drawing; In step S2, the characterization of indirect clue drawing includes: The size of the physical instance and the coordinates of the center point position are obtained according to the visual information of the physical instance, and the visual information of the physical instance is replaced with a preset virtual feature map to obtain a corresponding virtual object; The size and center point position coordinates of the preset virtual feature map are the same as those of the physical instance; In step S4, the intersection-and-union ratio between the segmentation bounding box A of the active movable object and the segmentation bounding box B of each passive movable object is calculated according to the following formula: :

[0043] in, The intersection of segmentation bounding box A and segmentation bounding box B is x The length of the side on the axis; The intersection of segmentation bounding box A and segmentation bounding box B is y The length of the side on the axis; is the coordinate of the lower left corner vertex of the segmentation bounding box A; is the coordinate of the upper right corner vertex of the segmentation bounding box A; is the coordinate of the lower left corner vertex of the segmentation bounding box B; is the coordinate of the upper right corner vertex of the segmentation bounding box B; In step S4, predicting the user's interaction target according to the calculation result includes: It is determined whether the intersection-and-union ratio between the segmentation bounding box of the active movable object and the segmentation bounding box of each passive movable object is greater than or equal to a preset threshold. If so, the category of the passive movable object is a target to be interacted with; if less than, the category of the passive movable object is a non-interactive target.

[0044] In the specific implementation process, the realization of synchronous mixed reality technology mainly includes four key steps: physical target information extraction, target characterization drawing, virtual-reality mapping and target selective spatiotemporal synchronization; the target information extraction process extracts the visual information and posture information of the target in the physical scene through image segmentation method; target characterization drawing extracts the feature information of virtual objects and assigns physical targets to achieve visual feature synchronization of virtual and real objects; virtual-reality mapping transforms the physical space coordinate system with the virtual space coordinate system to achieve target posture tracking; target selective spatiotemporal synchronization is to selectively express specific target information in the virtual environment according to user needs; In the physical target information extraction based on instance segmentation, the physical space information is first captured by the RGB camera and the deep learning instance segmentation algorithm is used to calculate the category, bounding box, and mask prototype information of the physical instance, and obtain its visual information and pose information. Secondly, the classification information is judged through the classification hash table and the physical target information is selectively extracted; In this embodiment, the extracted physical instances only retain active movable objects and passive movable objects, and for the extracted immovable objects, their pixel values ​​are directly set to 0; The segmentation algorithm in this embodiment is specifically the YOLACT instance segmentation model; the YOLACT model is a "first-order" instance segmentation model, which improves the algorithm speed by abandoning the feature localization step and adopting a bidirectional parallel approach; the YOLACT network structure is as follows Figure 6As shown in the figure, when a picture is input into the YOLACT network, the image is firstly feature extracted through the target detection network (RetinaNet) to obtain a set of feature layer images; secondly, the feature layer images are processed in two lines in parallel, one branch obtains the target mask, bounding box and category output through the mask coefficient prediction network and the non-maximum suppression algorithm (NMS); the other branch predicts the target prototype through the fully convolutional neural network (FCN) to obtain several prototype masks; finally, the mask prototype and mask coefficient of each instance are linearly combined to obtain the mask information of the instance and the output result of the image instance segmentation is obtained through bounding box cropping and thresholding. The YOLACT network uses a first-order RetinaNet target detection network to extract feature information of physical space target objects. The network consists of a residual subnetwork (RestNet-101) and a feature pyramid subnetwork (Feature Pyramid Network, FPN). Its network framework is as follows: Figure 7 As shown; YOLACT network selection Figure 8 The residual bottleneck block shown in the figure is used to construct a convolutional network with a depth of 101 layers to extract image features with more complex information; in the feature extraction process, the layer with unchanged feature map size during the convolution of the RestNet-101 network is taken as a stage, and the output result of the last layer of each stage is extracted as the feature pyramid layer. The outputs of the conv2, conv3, conv4 and conv5 stages are extracted, corresponding to the C2, C3, C4 and C5 layers of the feature pyramid. The convolution structure of each layer is shown in Table 1; Table 1 The convolutional layer structure of RestNet-101 residual subnetwork

[0045] The feature pyramid subnetwork FPN consists of three parts: bottom-up convolutional neural network, top-down feature sampling and lateral connection. The bottom-up convolutional network is RestNet-101 network, which can effectively extract coarse-grained features of the image. As the number of convolutional network layers increases, the target high-level semantic information becomes richer; however, as the convolution goes deeper, the image spatial resolution decreases and the spatial information loss increases. The top-down sampling process uses the interpolation method to extract feature information and obtains feature maps with weak high-level semantic information but rich spatial information; thereafter, the feature map with high semantic information and low spatial information obtained by bottom-up sampling is horizontally connected with the feature map with low semantic information and high spatial information obtained by top-down sampling, and the connection result is convolved with a 3×3 convolution kernel to eliminate the aliasing effect of upsampling and finally obtain a set of feature maps with both high spatial resolution and strong semantic information.

[0046] In the RetinaNet target detection network, a set of feature maps {C1, C2, C3, C4, C5} are first obtained by bottom-up sampling. In order to reduce the amount of calculation and improve the algorithm speed, the C1 feature layer with too low sampling step is discarded. Secondly, according to the feature pyramid network, the {C2, C3, C4, C5} feature layers are iterated layer by layer to obtain the {P2, P3, P4, P5} feature pyramid with both high spatial resolution and strong semantic information. Thereafter, in order to further improve the algorithm operation speed, the P2 feature layer with too high resolution is discarded. At the same time, in order to improve the accuracy of the algorithm, the P5 feature layer is strided with a step size of 2 to obtain the feature layer P6. The P6 feature layer is further strided to obtain the P7 feature layer and construct a feature pyramid containing the {P3, P4, P5, P6, P7} feature maps. In this feature pyramid, the deeper the feature layer, the higher the spatial resolution, and the higher the feature layer, the richer the semantic information. Finally, the feature layers {P3, P4, P5, P6, P7} are operated to extract the target feature information; The goal of the prototype generation network is to obtain a prototype mask with the highest possible resolution and the most accurate target information. Since the P3 layer has the highest resolution information in the feature pyramid generated by the RetinaNet target detection network, the P3 layer is used to generate the prototype mask. The prototype mask generation network is called the Protonet network, which is built based on the Fully Convolutional Network (FCN) and activated by the ReLU function. The Protonet network structure is as follows: Fig. 9 As shown in the figure, in the process of obtaining the prototype template, the input P3 feature map is firstly subjected to four identical convolution operations using a convolution kernel of size 3×3; secondly, the image after convolution is deconvolved using a deconvolution kernel of size 2×2 to enlarge the image size and improve the accuracy of small target segmentation; finally, a 1×1 convolution operation is performed on the upsampled image to obtain an image with K channels, and each channel of the image can be regarded as a prototype mask; The mask coefficient prediction network is used to predict the mask coefficients. The anchor detection model is often used for coefficient prediction. In the mask prediction process, it is first necessary to generate corresponding anchor points for each feature layer. In the YOLCAT algorithm, the translation-invariant anchor points are used to traverse the feature map and extract feature information. Fig.10 As shown, it includes a classification subnetwork, a box regression subnetwork, and a mask coefficient prediction subnetwork set in parallel; In the network pre-training stage, the corresponding loss function of the classification subnetwork is: classification loss function , calculated using Softmax cross entropy loss:

[0047]

[0048] in, is the true label, is the output value of the i-th node of the classification subnetwork, Output value Probability values ​​in C categories; Box regression loss function of the box regression subnetwork Includes: Confidence loss and position loss , and the two are weighted added:

[0049] Among them, x is the Ath i The prior box and the B j The matching degree of the bounding box; c is the category confidence prediction value; l is the prior box position prediction value; g is the bounding box position parameter; N is the number of prior box samples matched to the bounding box; To adjust and The proportional coefficient of the ratio between them is generally set to 1; Confidence loss Using Softmax cross entropy loss function,

[0050]

[0051] in, represents the positive sample set, represents the negative sample set, p represents the pth category, Represents the probability that the output value of the i-th node is in the p-th category; Position loss Using Smooth L1 loss calculation:

[0052]

[0053] in, Predict the predicted value of the corresponding bounding box for the i-th prior box; Match the encoding value of the jth real sample box to the i-th prior box; In the function Indicates the difference between the true value and the predicted value; The mask coefficient prediction subnetwork processes the feature map pixel by pixel and predicts a mask coefficient for each pixel i in the feature map. The mask coefficient prediction loss function is is the binary cross entropy loss function:

[0054]

[0055] The non-maximum suppression algorithm NMS in this embodiment is specifically a Fast NMS algorithm, which can reduce the impact of confidence score sorting on the algorithm running speed. It is not necessary to sort all the confidence output results, but only needs to extract the n results with the highest scores and calculate the IoU coefficient of the results; In the YOLCAT instance segmentation network, the algorithm running speed is improved by dividing the feature layer extracted by the RetinaNet target detection network into two branches for parallel processing; the prototype mask network processes the P3 feature layer through the Protonet network to obtain the prototype mask; the coefficient prediction network predicts the mask category, border and mask coefficient through the prediction head network, and uses the non-maximum suppression algorithm to obtain the local optimal solution of the mask coefficient; thereafter, a simple linear combination of the prototype mask and the mask coefficient can be used to obtain the final mask information; the mask information synthesis operation can be regarded as a combination of basic matrix multiplication and Sigmoid function, and the operation formula is as follows: The coefficient matrix and the prototype mask matrix Perform a combined operation according to the following formula to obtain the output of the YOLACT instance segmentation model :

[0056] in, is the Sigmoid function; When constructing a virtual-reality fusion environment, you first need to capture the following Fig.11 The physical space information shown in (a) is then segmented using the YOLCAT model to obtain the following: Fig.11 The physical object category, bounding box and mask prototype information shown in (b) are finally obtained through information extraction methods. Fig.11 The target visual feature information shown in (c) is shown in the figure. However, in real application scenarios, the physical environment contains a lot of useless information. Therefore, before virtual-real fusion, the spatial information segmented by the YOLCAT network needs to be processed to remove redundant information. This embodiment intelligently extracts target information by constructing a classified hash table. According to the system functions and user needs, the target object categories that need to be integrated into the virtual environment are stored in the hash table, and specific target information is extracted through category selection and judgment; in the actual application process, the categories in the list can be changed according to the different requirements of the system, so as to change the fused target information and realize the intelligent extraction of target information; The information extraction method combining the YOLCAT instance segmentation network and the classification hash table can selectively extract the visual information of the target object in the physical space and integrate it into the virtual environment to achieve virtual-real fusion. However, due to the large differences between most physical environments and virtual environments, directly integrating the extracted information into the virtual environment will destroy the environmental immersion. Therefore, in order to ensure the visual consistency of virtual information and physical information and maintain the immersion of the virtual-real fusion environment, target characterization drawing is required. In the process of target characterization drawing, it is necessary to first build a physical target label library according to the target category, then use the feature extraction method to calculate the virtual environment features and assign the virtual feature information to the physical target label to realize the feature presentation of the label library, and finally use the feature label to replace the physical information to realize the unification of the visual features of the physical target and the virtual environment, so as to ensure the immersion of the virtual and real fusion environment; The motion range and activity trajectory of active movable objects are uncontrollable, but the personalized features are obvious and personalized visual information is of great significance to the interaction. Therefore, direct graphical presentation is used to express information in the virtual environment; Fig.12 As shown in the figure, the direct clue drawing method draws the virtual agent pixel by pixel according to the contour, texture, color and other characteristics of the physical object; the physical information drawing includes not only the physical target drawing, but also the background information drawing. In the direct clue drawing scene, the pixel-by-pixel drawing method is used to draw the visual feature information of the physical target, and the redundant information is eliminated by setting the background or irrelevant information pixel value to 0, so as to realize the real-time rendering of the target information in the virtual environment; The direct clue drawing method can present the personalized characteristic information of the target and can respond to the changes of the target's visual characteristics in real time, which helps to improve the accuracy of system interaction. However, due to the limitation of information extraction, the information drawn by this method is slightly less accurate, and when the visual characteristics of the physical target are too different from the virtual environment, this method will destroy the immersion of the virtual-real fusion environment. Passive movable objects, whose motion trajectories are controllable and there are many of them, but their personalized visual features do not have specific meanings. Therefore, they can be replaced with characterized virtual substitutes in the virtual environment to maintain the immersive feeling of the environment while ensuring the interactive efficiency; indirect clue drawing replaces the characteristics of the target visual information according to the size of the target object and the coordinates of the center point position, and realizes the expansion of the target function or the visual unification of virtual and real information while retaining some physical object features; the indirect clue drawing process is as follows Fig.13As shown in the figure, in the information drawing process, the target information is no longer drawn pixel by pixel, but a pre-built virtual agent is used to indirectly express the physical object information. The texture, color, size, shape and other features of the virtual agent may be different from those of the physical object. This drawing method increases the consistency of virtual and real information and has less damage to the immersive feeling of the virtual and real fusion environment. However, due to the loss of target visual information, the system interaction accuracy is also low. Immovable objects: These objects have fixed positions and low interaction frequency, and can be left undrawn after matching the virtual space with the physical space. like Fig.14 , which is a schematic diagram of drawing different target object information visualization clues in this embodiment, directly drawing the active movable object (user's hand) and indirectly drawing the passive movable object (cup); In the virtual-reality mapping process, the camera coordinate system of the virtual space is converted into the camera coordinate system of the physical space through space transformation, so as to unify the position, direction and distance of the virtual space and the physical space, and complete the rendering of the virtual-reality fusion scene; For virtual-reality fusion scenes, the ideal state is: when there is no need to interact with an object, the object information is presented in the form of a characteristic icon to ensure the immersion of the virtual environment; when it is necessary to interact with such an object, the drawing mode of the target information is switched back to direct drawing to achieve accurate expression of physical information; to achieve this goal, this embodiment uses the intersection-over-union algorithm to predict the user's interaction purpose and presents the physical target visual information through a selective expression method; like Fig.15 As shown, the intersection-and-union ratio between the segmentation bounding box A of the active movable object and the segmentation bounding box B of each passive movable object is calculated according to the following formula: :

[0057] in, The intersection of segmentation bounding box A and segmentation bounding box B is x The length of the side on the axis; The intersection of segmentation bounding box A and segmentation bounding box B is y The length of the side on the axis; is the coordinate of the lower left corner vertex of the segmentation bounding box A; is the coordinate of the upper right corner vertex of the segmentation bounding box A; is the coordinate of the lower left corner vertex of the segmentation bounding box B; is the coordinate of the upper right corner vertex of the segmentation bounding box B; The larger the value, the higher the overlap; like Fig.16As shown in the figure, in the process of selective spatiotemporal synchronization of target information, the bounding box intersection-union algorithm is first used to predict the user's interaction purpose based on the target depth occlusion relationship. When the user does not need to interact with the object, the feature icon drawing method is used to express the physical information to ensure the immersion of the virtual-reality fusion environment. When the user needs to interact with the target, the interactive object is converted from the feature icon to the physical icon, while other non-interactive objects are still presented as feature icons, so as to balance the conflict between the immersion of the virtual environment and the interaction efficiency. After the physical scene information is segmented by YOLCAT instances, in addition to obtaining instances and instance classification labels, instance bounding box information is also obtained. In the process of selective information expression, the bounding box information output by the instance segmentation network is first used as the basis, and the intersection-union ratio between instances is calculated using the above formula. ; Secondly, set the threshold T and calculate Compared with the threshold T, if When ≥T, it indicates that the two borders intersect, and it is predicted that the instance is the user's interactive target; thereafter, the direct clue drawing method is used to express the target visual information, and the characteristic icon drawing method is used to indirectly express the non-interactive target visual information to achieve the selective expression of the target; In the intersection-and-union ratio calculation process, the first depth level object is used as the base bounding box to calculate the intersection-and-union ratio between the first depth level object bounding box and the second depth level object bounding box; Fig.16 In the figure, the first depth level object is the user's hand information. By calculating the intersection and union ratio of the user's hand bounding box with the bounding boxes of objects such as the cup, mouse, and keyboard, different scene drawing methods are selected; in the figure, the user's hand is on the keyboard, and the intersection of the keyboard bounding box and the hand bounding box is . The value is greater than the threshold T. At this time, the keyboard is presented in real time. However, the water cup bounding box and the hand bounding box do not overlap or the overlapping part is too small. If the value is less than the threshold, a characteristic icon drawing method is used; this method of selectively expressing information based on the predicted purpose of interaction can improve the system interaction efficiency while maintaining the immersiveness of the virtual environment.

[0058] Example 3 This embodiment provides a synchronous mixed reality hardware system, based on a synchronous mixed reality method with both immersion and interactive performance described in Embodiment 1 or 2.

[0059] In the specific implementation process, the synchronous mixed reality hardware system performs virtual-real fusion in units of space, which can be divided into small-scale specific space virtual-real fusion and large-scale movable space virtual-real fusion according to the fusion space range; the two types of space virtual-real fusion have different requirements for information capture field of view. When performing small-scale close-range physical object fusion, this type of space generally contains immovable objects and passive movable objects, and has low requirements for field of view, but high requirements for information extraction resolution, so a high-resolution monocular camera is used for construction; when performing large-scale long-range physical object fusion, this type of space contains three types of interactive characteristic objects, and has high requirements for field of view, so a binocular camera is used for construction; The two systems have similar structures, both of which include an information capture and calculation unit, an information calculation and rendering unit, and an information visualization unit, wherein the information capture unit completes the real-time capture of physical space information, mainly a binocular or monocular camera; the information calculation and rendering unit is used to complete the real-time segmentation, extraction and rendering of spatial information using the method in Example 1 or 2, and a desktop computer is used for processing; the information visualization unit is used to present real-time information of virtual-real fusion scenes, mainly a head-mounted display device; like Fig.17 As shown in FIG. 1 , it is a schematic diagram of scene rendering using a synchronous mixed reality hardware system, such as Fig.17 The physical environment in (a) includes a table and several ceramic cups with different textures and colors. The hardware used by the user is a monocular camera and an HTC VIVE head-mounted display device; the virtual environment is as follows: Fig.17 As shown in (b), a table corresponding to the physical space is drawn to place the extracted cup model; Fig.17 (c) shows a synchronous mixed reality scene drawn with direct cues. Fig.17 (d) in the figure shows the selective presentation effect of the characteristic image drawing scene and the target information; it can be seen that the system can successfully realize the selective expression of the target information; in addition, experimental verification using the system shows that the physical information selective expression method proposed in Example 1 or 2 can effectively solve the conflict between the environmental presence and the system interaction efficiency in the virtual-reality fusion scene drawing, and can improve the virtual environment immersion by 15.3%, increase the interaction speed by 14.6% and reduce the number of collisions by 34.3%. On the basis of ensuring the environmental immersion, the target interaction accuracy is improved, and the environmental performance and system interaction efficiency can be taken into account. At the same time, the rendering freedom of the virtual-reality fusion scene is expanded, and the intelligent fusion of virtual and real space is realized.

[0060] The same or similar reference numerals correspond to the same or similar components; The terms used to describe the positional relationship in the drawings are only used for illustrative purposes and should not be construed as limiting the present application; Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A synchronous mixed reality method with both immersion and interactive performance, characterized in that: The following steps are involved: S1: Use a camera to acquire a physical scene image, and use a pre-trained instance segmentation model to extract visual information, pose information, and segmentation bounding boxes of several physical instances from the physical scene image; The physical instances include: active movable objects and passive movable objects; S2: Load the virtual environment, draw direct clues for the visual information of the active movable object, draw characteristic indirect clues for the visual information of the passive movable object, and obtain the virtual object corresponding to each physical instance; S3: Obtain the position and posture information of each virtual object in the virtual environment, and map the virtual and real coordinates with the position and posture information of the corresponding physical instance to complete the rendering of the virtual and real fusion scene; S4: Based on the rendered virtual-reality fusion scene, calculate the intersection-over-union ratio between the segmentation bounding box of the active movable object and the segmentation bounding box of each passive movable object, predict the user's interaction target according to the calculation result, and classify all passive movable objects into targets to be interacted and non-interaction targets; S5: redrawing the visual information of the target to be interacted with by direct clue drawing, obtaining the secondary drawn virtual object corresponding to the target to be interacted with and performing visual expression; for the virtual objects corresponding to the active movable objects and non-interactive targets, performing visual expression directly; S6: Repeat steps S4 to S6 to complete the synchronous mixed reality of selectively expressing the target information.

2. A synchronous mixed reality method with both immersion and interactive performance according to claim 1, characterized in that: In step S1, the pre-trained instance segmentation model is specifically: a YOLACT instance segmentation model; The YOLACT instance segmentation model includes: a target detection network RetinaNet, and a mask coefficient prediction network and a fully convolutional neural network FCN set in parallel; the output of the target detection network RetinaNet is connected to the input of the mask coefficient prediction network and the fully convolutional neural network FCN respectively; The output of the mask coefficient prediction network is processed by the non-maximum suppression algorithm NMS, and then combined with the output of the fully convolutional neural network FCN to obtain the output of the YOLACT instance segmentation model.

3. A synchronous mixed reality method with both immersion and interactive performance according to claim 2, characterized in that: In the YOLACT instance segmentation model, the target detection network RetinaNet includes a RestNet-101 residual subnetwork and a feature pyramid subnetwork FPN connected in sequence; The mask coefficient prediction network includes a classification subnetwork, a frame regression subnetwork and a mask coefficient prediction subnetwork arranged in parallel; The fully convolutional neural network FCN is specifically a Protonet prototype mask generation subnetwork; The non-maximum suppression algorithm NMS is specifically a Fast NMS algorithm.

4. A synchronous mixed reality method with both immersion and interactive performance according to claim 2, characterized in that: The output of the mask coefficient prediction network is processed by the non-maximum suppression algorithm NMS to obtain The coefficient matrix of , where n is the number of physical instances calculated by the non-maximum suppression algorithm NMS, and k is the mask coefficient of each physical instance; The output of the fully convolutional neural network FCN is The prototype mask matrix , where h and w are the first and second sizes of the prototype mask, respectively. The number of prototype masks for each physical instance; The coefficient matrix and the prototype mask matrix Perform a combined operation according to the following formula to obtain the output of the YOLACT instance segmentation model : in, is the Sigmoid function.

5. A synchronous mixed reality method with both immersion and interactive performance according to claim 2, characterized in that: The loss functions of the YOLACT instance segmentation model in the pre-training process include: classification loss function , box regression loss function And the mask coefficient prediction loss function ; The classification loss function Specifically, it is the Softmax cross entropy loss function; The box regression loss function Includes: Confidence loss and position loss , the confidence loss Specifically, it is the Softmax cross entropy loss function, the position loss Specifically, it is the Smooth L1 loss function; The mask coefficient prediction loss function Specifically, it is a binary cross entropy loss function.

6. A synchronous mixed reality method with both immersion and interactive performance according to claim 1, characterized in that: The step S1 further includes: setting a category hash table, wherein the category hash table stores several categories of physical instances that can be integrated into the virtual environment; Determine whether each extracted physical instance belongs to the physical instance category in the category hash table. If so, directly execute step S2; if not, delete the visual information, posture information and segmentation bounding box of the physical instance and execute step S2.

7. A synchronous mixed reality method with both immersion and interactive performance according to claim 1, characterized in that: In the step S2 and step S5, direct clue drawing includes: physical instance body drawing and background information drawing; The physical instance ontology drawing specifically includes: drawing the outline, texture and color of the physical instance pixel by pixel in the virtual space according to the visual information of the physical instance, and obtaining the corresponding virtual object; The background information drawing is specifically as follows: taking pixels other than the pixels of the physical instance as background pixels, setting the pixel values ​​of the background pixels to 0, and completing the background information drawing.

8. The synchronous mixed reality method with both immersion and interactive performance according to claim 1, characterized in that: In step S2, the characterization of indirect clue drawing includes: The size of the physical instance and the coordinates of the center point position are obtained according to the visual information of the physical instance, and the visual information of the physical instance is replaced with a preset virtual feature map to obtain a corresponding virtual object; The size and center point position coordinates of the preset virtual feature map are the same as those of the physical instance.

9. The synchronous mixed reality method with both immersion and interactive performance according to claim 1, characterized in that: In step S4, the intersection-and-union ratio between the segmentation bounding box A of the active movable object and the segmentation bounding box B of each passive movable object is calculated according to the following formula: : in, The intersection of segmentation bounding box A and segmentation bounding box B is x The length of the side on the axis; The intersection of segmentation bounding box A and segmentation bounding box B is y The length of the side on the axis; is the coordinate of the lower left corner vertex of the segmentation bounding box A; is the coordinate of the upper right corner vertex of the segmentation bounding box A; is the coordinate of the lower left corner vertex of the segmentation bounding box B; is the coordinate of the upper right corner vertex of the segmentation bounding box B.

10. The synchronous mixed reality method with both immersion and interactive performance according to claim 1, characterized in that: In step S4, predicting the user's interaction target according to the calculation result includes: It is determined whether the intersection-and-union ratio between the segmentation bounding box of the active movable object and the segmentation bounding box of each passive movable object is greater than or equal to a preset threshold. If so, the category of the passive movable object is a target to be interacted with; if less than, the category of the passive movable object is a non-interactive target.

Images