Augmented reality method and apparatus, and storage medium and electronic device

Abstract

Disclosed in the present disclosure are an augmented reality method and apparatus, and a storage medium and an electronic device. A plurality of modules which can cooperate with each other are deployed for an augmented reality device, such that a user using the augmented reality device can use the augmented reality device to execute and complete virtual and real interaction; moreover, an operation performed by the user when executing an interaction service can be synchronized to other users, such that the user can share virtual and real interaction with the other users.

Description

Augmented reality method, device, storage medium and electronic device Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, storage medium, and electronic device for augmented reality. Background Art

[0002] Augmented / mixed reality technology seamlessly overlays computer-simulated digital information data with real-world environments, breaking through spatial and temporal constraints to achieve highly immersive, seamless real-time information guidance and virtual-reality interaction. Its essence lies in "what you see is what you get, what you touch is what you see." With this technology, information displayed / interacted on screens can be replaced with three-dimensional information that is strongly associated with objects in the actual environment and presented intuitively to the user. This means that users can use any surface in front of them as a screen and place any virtual object in any field of view (arranged just like real objects, even simulating physical properties). This breaks down the barriers between virtual and real space and allows for direct communication between virtual and real spaces. This means that through technologies such as gesture recognition, depth recognition, and artificial intelligence, user intent is captured and understood, eliminating the disconnect between the digital virtual space and the real environment, allowing information exchange to be completed through the user's own efforts.

[0003] Because augmented / mixed reality devices and system applications are still in the development stage, and there are no unified standards or methods for general application framework design, this paper proposes a universal application system framework from the perspectives of application system architecture, module design, and multi-person collaboration to meet the practical application needs of future augmented / mixed reality environments.

[0004] Summary of the Invention

[0005] The present disclosure provides an augmented reality method, device, storage medium, and electronic device to partially solve the above-mentioned problems existing in the prior art.

[0006] This disclosure adopts the following technical solutions:

[0007] The present disclosure provides an augmented reality device, wherein the device is provided with a data-driven interactive interface module, a universal and extensible service module, an identification and tracking module, a multi-person collaboration module, and a hybrid network communication module;

[0008] The data-driven interactive interface module is configured to instantiate, for each interface element included in an interactive interface for interacting with a user, a class corresponding to the interface element to obtain a first instantiated object, and instantiate an element data class pre-bound to the interface element to obtain a second instantiated object, and for each specified variable included in the first instantiated object, bind the specified variable to a specified variable with the same name as the specified variable among the specified variables included in the second instantiated object, so that when the value of any specified variable bound to the first instantiated object in the second instantiated object changes, the presentation of the interface element corresponding to the specified variable bound to the specified variable in the first instantiated object in the interactive interface changes synchronously;

[0009] The universal extensible service module is configured to pre-store containers corresponding to services executed by the augmented reality device and, when executing a target service, call components in the container corresponding to the target service to execute the target service based on the called components; create a new service to be expanded using the preset components and manage the creation status of the service to be expanded;

[0010] The recognition and tracking module is used to perform image recognition on the image data collected by the augmented reality device to determine the position and posture of the target object in the augmented reality interaction scene where the user is located;

[0011] The multi-person collaboration module is configured to respond to a data synchronization instruction and send the data synchronization instruction generated by the local device to other augmented reality devices, wherein the data synchronization instruction is used to control a target object in the augmented reality interaction scene or update the state of the target object;

[0012] The hybrid network communication module is used to determine a transmission strategy according to different types of data to be transmitted generated in the augmented reality interaction scene, and transmit different types of data to be transmitted according to different transmission strategies.

[0013] Optionally, the data-driven interactive interface module is further used to update the elements in the interface in real time according to the update status of the back-end data bound to the interface elements in the display interface of the augmented reality device.

[0014] The present disclosure also provides a method for augmenting reality, comprising:

[0015] Creating an augmented reality interaction scene based on the augmented reality interaction scene creation request received from the user;

[0016] According to element variables preset for interface elements and data variables preset for element data, for each interface element, the element variable of the interface element is bound to the data variable of the element data required for rendering the interface element, thereby implementing binding of the interface element with the element data corresponding to the interface element, so that the interface element affected by the user's interactive action and the element data required for rendering the interface element are bound in the augmented reality interactive scene through the defined designated object class, wherein the designated object class includes an object class corresponding to the interface element and an object class corresponding to the interface element data;

[0017] When an interactive action performed by the user is monitored, the augmented reality interaction scene to be displayed by the user in the augmented reality device is determined, and the business scene data required in the augmented reality interaction scene and the common data used in different augmented reality interaction scenes are called to display the virtual interface object to be displayed by the user in the display interface of the augmented reality device, and the position and posture of the target object in the augmented reality interaction scene in which the user is located are determined;

[0018] The posture and the synchronization instruction corresponding to the interactive action performed by the user are sent to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interaction scene presented by the augmented reality device according to the posture and the synchronization instruction.

[0019] Optionally, the method further includes: updating the elements in the interface in real time according to the update status of the data bound to the interface elements in the display interface of the augmented reality device in the back end.

[0020] Optionally,

[0021] Determining the position and posture of a target object in the augmented reality interaction scene in which the user is located includes:

[0022] Collecting image data of the augmented reality scene;

[0023] Identifying, according to a preset recognition model, a posture of the target object in the image represented by the image data as an initial posture;

[0024] The posture of a target object in the augmented reality interaction scene where the user is located is determined based on the initial posture and the image data.

[0025] Optionally,

[0026] Sending the posture and the synchronization instruction corresponding to the interactive action performed by the user to other augmented reality devices, including:

[0027] Determining scene data corresponding to the posture and the synchronization instruction corresponding to the interactive action performed by the user in the augmented reality interaction scene;

[0028] The scene data is transmitted to other augmented reality devices.

[0029] Optionally, the scene data is transmitted to another augmented reality device, and the method further includes:

[0030] Obtain the data transmission policy determined by the user;

[0031] The scene data is transmitted to other augmented reality devices according to the data transmission strategy determined by the user.

[0032] Optionally, the method further includes:

[0033] Obtain new business demand instructions sent by users;

[0034] In response to the newly added service requirement instruction, the service to be expanded is determined, the service to be expanded is instantiated through a specified container, and the executable service in the augmented reality device is updated according to the instantiated service to be expanded.

[0035] The present disclosure provides an augmented reality device, comprising:

[0036] A creation module, configured to create an augmented reality interaction scene based on an augmented reality interaction scene creation request received from a user;

[0037] A binding module is used to pre-set an element variable for each interface element and a data variable for each element data, and for each interface element, bind the element variable of the interface element to the data variable of the element data required for rendering the interface element, thereby binding the interface element to the element data corresponding to the interface element, so that the interface element affected by the user's interactive action and the element data required for rendering the interface element are bound in the augmented reality interaction scene through the defined specified object class, wherein the specified object class includes an object class corresponding to the interface element and an object class corresponding to the interface element data;

[0038] A determination and display module, configured to, when detecting an interactive action performed by a user, determine an augmented reality interaction scene to be displayed by the user in the augmented reality device, and call business scenario data required in the augmented reality interaction scene and common data used in different augmented reality interaction scenes to display the virtual interface object to be displayed by the user in the display interface of the augmented reality device, and determine the position and posture of the target object in the augmented reality interaction scene in which the user is located;

[0039] A synchronization module is used to send synchronization instructions corresponding to the posture and the interactive action performed by the user to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interaction scene presented by the augmented reality device according to the posture and the synchronization instructions.

[0040] Optionally, the device further includes:

[0041] An updating module is used to update the elements in the display interface of the augmented reality device in real time according to the update status of the data bound to the interface elements in the back end.

[0042] Optionally, the determination and display module is used to collect image data of the augmented reality scene; identify the posture of the target object in the image represented by the image data as an initial posture according to a preset recognition model; and determine the posture of the target object in the augmented reality interaction scene in which the user is located based on the initial posture and the image data.

[0043] Optionally, the synchronization module is used to determine scene data corresponding to the posture and the synchronization instruction corresponding to the interactive action performed by the user in the augmented reality interaction scene; and transmit the scene data to other augmented reality devices.

[0044] Optionally, the synchronization module is further used to obtain a data transmission strategy determined by a user; and transmit the scene data to other augmented reality devices according to the data transmission strategy determined by the user.

[0045] Optionally, the device further includes:

[0046] The business expansion module is used to obtain the new business demand instruction sent by the user; in response to the new business demand instruction, determine the business to be expanded, instantiate the business to be expanded through a specified container, and update the executable business in the augmented reality device according to the instantiated business to be expanded.

[0047] The present disclosure provides a computer-readable storage medium storing a computer program, which implements the above-mentioned augmented reality method when executed by a processor.

[0048] The present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned augmented reality method when executing the program.

[0049] At least one of the above technical solutions adopted in the present disclosure can achieve the following beneficial effects:

[0050] The augmented reality method provided by the present disclosure creates an augmented reality interaction scene based on an augmented reality interaction scene creation request received from a user; by presetting element variables for each interface element and data variables for each element data, for each interface element, the element variable of the interface element is bound to the data variable of the element data required for rendering the interface element, thereby realizing the binding of the interface element with the element data corresponding to the interface element, so that the interface elements affected by the user's interactive action and the element data required for rendering the interface element are bound in the augmented reality interaction scene through the defined specified object class, wherein the specified object class includes the object class corresponding to the interface element, the interface element data corresponding object class; when an interactive action performed by a user is monitored, the augmented reality interactive scene to be displayed by the user in the augmented reality device is determined, and the business scenario data required in the augmented reality interactive scene and the common data used in different augmented reality interactive scenes are called to display the virtual interface object to be displayed by the user in the display interface of the augmented reality device, and the posture of the target object in the augmented reality interactive scene in which the user is located is determined; the posture and the synchronization instruction corresponding to the interactive action performed by the user are sent to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interactive scene presented by the augmented reality device according to the posture and the synchronization instruction.

[0051] It can be seen from the above method that by deploying a plurality of modules that can collaborate with each other for the augmented reality device, the user using the augmented reality device can use the augmented reality device to perform the interaction between virtual and reality, and the operations performed by the user when performing the interactive service can be synchronized to other users, so that the user can share the interaction between virtual and reality with other users. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute improper limitations on the present disclosure.

[0053] FIG1 is a schematic structural diagram of an augmented reality device provided in the present disclosure;

[0054] FIG2 is a flow chart of an augmented reality method provided in the present disclosure;

[0055] FIG3 is a schematic diagram of a process for creating a service to be expanded for an augmented reality device provided in the present disclosure;

[0056] FIG4 is a schematic diagram of a flow chart of information interaction provided in the present disclosure;

[0057] FIG5 is a schematic diagram of a process for real-time sharing of interactive data of an augmented reality device provided in the present disclosure;

[0058] FIG6 is a schematic diagram of a process of transmitting data by an augmented reality device provided in the present disclosure;

[0059] FIG7 is a schematic diagram of an augmented reality device structure provided by the present disclosure;

[0060] FIG8 is a schematic structural diagram of an electronic device provided by the present disclosure corresponding to FIG1 . DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the specific embodiments of the present disclosure and the corresponding drawings. The described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0062] The technical solutions provided by various embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0063] Figure 1 is a structural diagram of an augmented reality device provided in the present disclosure. The augmented reality device may include: a data-driven interactive interface module 101, a universal extensible business module 102, an identification and tracking module 103, a multi-person collaboration module 104, and a hybrid network communication module 105.

[0064] People can use the display interface of the augmented reality device in the embodiment of the present disclosure to observe virtual objects in the real environment, and perform operations on the target objects together with other users to interact, thereby establishing a connection between the virtual space and the real space.

[0065] The augmented reality device in the embodiment of the present disclosure can be applied to augmented reality interaction scenarios such as augmented reality guidance scenarios and augmented reality game scenarios.

[0066] For example, in a pipeline maintenance guidance scenario in an augmented reality guidance scenario, user A can see the pipeline that needs to be repaired in front of him through an augmented reality device such as AR (Augmented Reality) glasses held by him, and send the image of the pipeline in front of user A to user B through the AR glasses. User B can mark a virtual arrow in the video through a mobile terminal or augmented reality device such as a mobile phone or computer held by user B that can receive images and videos to guide user A to perform pipeline maintenance. User A can receive the virtual arrow marked by user B through the AR glasses held by user A himself, and user A can observe the virtual arrow marked by user B appearing in the image of the pipeline in front of him through the AR glasses held by user A himself, and perform relevant operations according to the instructions of the arrow.

[0067] During the operation of the augmented reality device, the augmented reality function of the augmented reality device can be realized through the following modules.

[0068] The data-driven interactive interface module 101 is used to instantiate the class corresponding to each interface element contained in the interactive interface for interacting with the user to obtain a first instantiated object, and instantiate the element data class pre-bound to the interface element to obtain a second instantiated object, and for each specified variable contained in the first instantiated object, bind the specified variable to a specified variable with the same name as the specified variable among the specified variables contained in the second instantiated object, so that when the value of any specified variable bound to the first instantiated object in the second instantiated object changes, the presentation of the interface element corresponding to the specified variable bound to the any specified variable in the first instantiated object in the interactive interface changes synchronously.

[0069] The class corresponding to the above-mentioned interface element can be an abstract interface element parent class, and the element data class bound to the interface element can be an abstract element data parent class. According to the user's interactive action in the augmented reality interaction scene, the interface element corresponding to the interactive action (for example, a virtual button) is determined, and the abstract interface element parent class is concretized as the interface element affected by the interactive action to obtain a specific interface element class. In addition, the abstract element data parent class is concretized as the background data of the interface element affected by the interactive action (i.e., the element data required for rendering the interface element) to obtain a specific element data class. By binding the specific interface element class with a preset binding element tool, binding the specific element data class with a preset binding element attribute, establishing a corresponding relationship between the binding element tool and the binding element attribute, and by changing the binding element attribute and the binding element tool, the specific interface element class and the specific element data class are adjusted to achieve binding of the interface elements affected by the user's interactive action and the element data required for rendering the interface elements in the augmented reality interaction scene.

[0070] The general extensible business module 102 is used to pre-save the container corresponding to the business executed by the augmented reality device, and when executing the target business, call the components in the container corresponding to the target business to execute the target business based on the called components; create a new business to be expanded through the preset components, and manage the creation status of the business to be expanded.

[0071] Specifically, the augmented reality device may first obtain a new business demand instruction sent by the user or a new business demand instruction sent by the server, and then respond to the new business demand instruction to determine the business to be expanded, instantiate the business to be expanded through a specified container, and update the executable business in the augmented reality device according to the instantiated business to be expanded.

[0072] The recognition and tracking module 103 is used to perform image recognition on the image data collected by the augmented reality device to determine the position and posture of the target object in the augmented reality interaction scene where the user is located.

[0073] The augmented reality device can specifically identify the target object based on the image (including color image and depth image) captured by the camera installed in the device through a preset recognition model, and identify the initial posture of the target object in the image (the posture includes position and direction), and based on the image, the initial posture and the parameters of the internal and external matrices of the camera installed in the device, determine the actual posture of the target object in the augmented reality interaction scene where the user is located.

[0074] The multi-person collaboration module 104 is used to respond to data synchronization instructions and send data synchronization instructions generated by the local device to other augmented reality devices. The data synchronization instructions are used to control the target object in the augmented reality interaction scene or update the state of the target object.

[0075] The augmented reality device can complete real-time information interaction between users based on the multi-person collaboration module.

[0076] The hybrid network communication module 105 is configured to determine a transmission strategy for different types of data to be transmitted generated in the augmented reality interaction scene, and transmit the different types of data to be transmitted according to different transmission strategies.

[0077] The augmented reality device can allocate a corresponding matching transmission strategy to the data to be transmitted based on the hybrid network communication module to achieve efficient data transmission.

[0078] Furthermore, the data-driven interactive interface module 101 is further configured to update the elements in the display interface of the augmented reality device in real time according to the update of the back-end data bound to the interface elements.

[0079] In addition, the augmented reality device may further include: a launcher 106, a spatial positioning module 107, a data storage module 108, and a state management module 109. The launcher 106 is used to launch the augmented reality device; the spatial positioning module 107 is used to determine the spatial positioning information of the augmented reality device; the data storage module 108 is used to store data generated during the execution of the service by the augmented display device; and the state management module 109 is used to manage the state of each of the above modules.

[0080] Based on the above-mentioned augmented reality device as shown in FIG1 , the present disclosure further provides a corresponding augmented reality method, as shown in FIG2 . FIG2 is a flow chart of an augmented reality method provided in the present disclosure, specifically including the following steps S201 - S204 .

[0081] S201: Creating an augmented reality interaction scene based on an augmented reality interaction scene creation request received from a user.

[0082] In the embodiment of the present disclosure, the method shown in FIG2 may be executed by the augmented reality device shown in FIG1 above.

[0083] Once the augmented reality interaction scene creation request sent by the user is received, the augmented reality device may create an augmented reality interaction scene corresponding to the augmented reality interaction scene creation request sent by the user.

[0084] Here, the augmented reality interaction scene creation request sent by the user may refer to the user touching a control in the augmented reality device interface that indicates a request for an augmented reality interaction scene, or it may refer to the user uttering a voice that indicates a request for an augmented reality interaction scene. In the present disclosed embodiment, no specific limitation is made on the triggering conditions of the above requests.

[0085] The augmented reality interaction scene created by the augmented reality device here refers to a scene consistent with the service corresponding to the augmented reality interaction scene creation request issued by the user. The augmented reality interaction scene can be a scene observed by the user through the augmented reality device.

[0086] For example, the user's real environment contains a real table, on which lies a real cup and five real cup lids. When the user sends an augmented reality interaction scene creation request to an augmented reality device such as AR glasses, indicating that the business the user wants to perform is a "multi-person cooperative cup lid twisting game," the AR glasses can respond to the request and create an augmented reality interaction scene corresponding to the business. The augmented reality interaction scene includes the above-mentioned real environment and a virtual arrow created by the AR glasses to guide the user to complete the game. The arrow can point to the cup lid that best matches the cup.

[0087] Subsequently, the user can interact with other users in the augmented reality interactive scene created by the augmented reality device.

[0088] S202: According to the element variables preset for the interface elements and the data variables preset for the element data, for each interface element, the element variables of the interface element are bound to the data variables of the element data required for rendering the interface element, so as to realize the binding of the interface element and the element data corresponding to the interface element, so that the interface elements affected by the user's interactive actions and the element data required for rendering the interface elements in the augmented reality interaction scene are bound through the defined designated object class, wherein the designated object class includes the object class corresponding to the interface element and the object class corresponding to the interface element data.

[0089] The augmented reality device will preset element variables for each interface element in the display interface of the augmented reality device, and for each of the above interface elements, bind the element variable of the interface element to the data variable of the element data required to render the interface element, so as to realize the binding of the interface element and the element data corresponding to the interface element, and through the pre-defined designated object class, in the augmented reality interaction scene, the interface elements affected by the user's interactive action and the element data required to render the interface element (that is, the data corresponding to the interface element in the background) are bound.

[0090] The designated object class mentioned here may include: an abstract interface element parent class (ie, the class corresponding to the above interface element) and an abstract element data parent class (ie, the element data class bound to the above interface element).

[0091] The abstract interface element parent class contains multiple different interface elements, and the abstract element data parent class contains the data corresponding to multiple different interface elements in the background.

[0092] Interface elements refer to the elements displayed on the display interface of an augmented reality device (such as buttons, charts, 3D models, and other elements in the interactive interface). Element data (the data corresponding to the interface element in the background) refers to the data representation corresponding to the above interface elements in the background (such as the trigger signal data of the button, the value data of the chart, the posture data of the 3D model, etc.).

[0093] The augmented reality device will pre-set corresponding binding element tools, such as variable A, for the interface elements affected by the user's interactive actions, and will also pre-set corresponding binding element attributes, such as variable B, for the background data of the interface elements affected by the user's interactive actions. In addition, the augmented reality device will also establish a corresponding relationship between the binding element tool and the binding element attribute.

[0094] Specifically, once the augmented reality device detects an interactive action initiated by the user in the augmented reality interactive scene, it can determine the interface element affected by the interactive action and the background data of the interface element affected by the user's interactive action. Furthermore, the abstract interface element parent class is concretized to the interface element affected by the interactive action, thereby obtaining a specific interface element class, and the abstract element data parent class is concretized to the background data of the interface element affected by the user's interactive action, thereby obtaining a specific element data class.

[0095] Afterwards, the specific interface element class can be bound to a preset binding element tool, and the specific element data class can be bound to a preset binding element attribute. In this way, after the augmented reality device detects the user's interactive action in the augmented reality interactive scene, the interface element affected by the interactive action can be determined as the binding element tool, and based on the correspondence between the binding element tool and the binding element attribute, the binding element attribute corresponding to the binding element tool can be determined as the background data corresponding to the interface element affected by the interactive action, that is, the specific element data class corresponding to the binding element tool.

[0096] For example, when a user makes an upward translation interaction action on a virtual triangle button in an augmented reality interaction scene, the interface element affected by the interaction action is the triangle button, that is, the specific interface element class is the triangle button. Since the interface element class is bound to the binding element tool variable A, the triangle button corresponding to variable A also performs the upward translation interaction action. Since the binding element attribute corresponding to the binding element tool variable A is variable B, the coordinates in the data corresponding to variable B will also be translated upward accordingly. Since the binding element attribute variable B is bound to the specific element data class corresponding to the triangle button, the coordinates in the data corresponding to the specific element data class corresponding to the triangle button will also be translated upward accordingly.

[0097] In addition, the elements in the interface can be updated in real time according to the update of the data bound to the interface elements in the display interface of the augmented reality device in the back end. The specific update process will not be repeated in this disclosure.

[0098] In this way, the interface elements displayed in the user's augmented reality device are dynamically bound to the backend data corresponding to the interface elements.

[0099] S203: When an interactive action performed by a user is monitored, the augmented reality interaction scene that the user wants to display in the augmented reality device is determined, and the business scenario data required in the augmented reality interaction scene and the common data used in different augmented reality interaction scenes are called to display the virtual interface object that the user wants to display in the display interface of the augmented reality device, and determine the posture of the target object in the augmented reality interaction scene where the user is located.

[0100] Among them, the business scenario data required in the augmented reality interaction scenario here refers to the data required for a specific business scenario, which may include data corresponding to rendering a specific augmented reality business scenario, and may also include data corresponding to different states of the same virtual object presented in different augmented reality interaction scenarios.

[0101] The interactive action here is used to represent the action initiated by the user based on the augmented reality device to execute the service selected by the user.

[0102] The user can touch the control used to create the virtual object, and this touch action can be used as an interactive action. Since the state of the same virtual object in different augmented reality interaction scenes may be different in some parts, but the other parts are the same. Therefore, in order to improve the efficiency of creating virtual objects, the augmented reality device can call the business scenario data required in the augmented reality interaction scene (that is, the data corresponding to the different states of the same virtual object in different augmented reality interaction scenes) and the general data used in different augmented reality interaction scenes (that is, the data corresponding to the same state of the same virtual object in different augmented reality interaction scenes) to specifically display the virtual interface object that the user wants to display in the display interface of the augmented reality device and determine the position and posture of the target object in the augmented reality interaction scene where the user is located.

[0103] For example, when creating a virtual object of a long-haired girl, the hair of the long-haired girl hangs down on land, but floats in space. However, the performance of the same virtual object of the long-haired girl in these two augmented reality interaction scenarios is consistent in all parts except the hair. Therefore, when creating the long-haired girl in the augmented reality interaction scenario of space, the data required to create the hair is the business scenario data mentioned above, while the data required to create other parts of the long-haired girl is the general data mentioned above.

[0104] Specifically, once the augmented reality device detects that the user has performed an interactive action, the augmented reality device can determine the virtual interface object to be displayed in the display interface of the augmented reality device. Afterwards, the augmented display device can determine the augmented reality interaction scene that the user wants to display in the augmented reality device, call the business scenario data required in the augmented reality interaction scene and the common data used in different augmented reality interaction scenes, and call the function in the preset object container that can be used to create the virtual interface object in the augmented reality interaction scene to create the virtual interface object, so as to display the virtual interface object that the user wants to display in the display interface of the augmented reality device.

[0105] In addition, since when users use augmented reality devices to perform business, it involves the user's operation on the target object displayed on the display interface of the augmented reality device, it is necessary to track the target object so that the user can successfully complete the operation on the target object or virtual interface object displayed on the display interface of the augmented reality device. At this time, the augmented reality device needs to determine the position of the target object in the augmented reality interaction scene where the user is located.

[0106] The target object here refers to the object that needs to be operated corresponding to the business performed by the user.

[0107] Continuing with the above example, when a user executes the "multi-player cooperative cup lid screwing game" through an augmented reality device, since the user selects the service of finding a matching cup lid for the cup and screwing the cup lid onto the cup, the target object at this time is the cup and a cup lid that matches the cup, and the virtual arrow guiding the user to complete the game is the virtual interface object to be displayed in the display interface of the augmented reality device.

[0108] When the augmented reality device determines the posture of the target object in the augmented reality interactive scene where the user is located, the augmented reality device may specifically capture an image of the augmented reality scene created for the user by the augmented reality device in front of the user through a camera installed in the device. The image here may include an RGB (Red, Grean, Blue) image of the image, i.e., a color image, and a depth image of the image. In addition, the image data corresponding to the image may also be determined. The image data here may include: the image data of the image and the depth data of the image, as well as identifying the posture of the target object in the pixel coordinate system of the image according to a preset recognition model as the initial posture. In addition, the preset recognition model here can also identify what the target object is specifically. The posture mentioned above includes position and direction.

[0109] Afterwards, the augmented reality device can determine the actual posture of the target object in the augmented reality interaction scene where the user is located based on the initial posture, the RGB image and depth image corresponding to the image, the above-mentioned image data and the parameters of the internal and external matrices of the camera installed in the device.

[0110] Specifically, a series of spatial coordinate transformations are performed to determine the actual position of the target object in the augmented reality interaction scene where the user is located. That is, first, a pixel-level one-to-one matching is performed based on the depth image and the RGB image to convert the pixel coordinate system where the initial position is located into the image coordinate system, and the initial position is also converted into the initial position in the image coordinate system. Thereafter, the augmented reality device can convert the image coordinate system into a camera coordinate system with the camera under the user's perspective (the position of the camera under the user's perspective is the same as that of the camera installed in the device) as the zero position, and convert the initial position in the image coordinate system into the initial position in the camera coordinate system. Afterwards, the camera coordinate system can also be converted into a world coordinate system based on the parameters of the internal and external matrices of the camera installed in the device, and the initial position in the camera coordinate system is converted into the initial position in the world coordinate system. The initial position in the world coordinate system here is the actual position of the target object in the augmented reality interaction scene where the user is located.

[0111] The preset recognition model can be constructed and trained according to the following ideas:

[0112] To achieve object recognition based on RGB images, we can first build an object recognition model framework: First, set a predefined scale range and center location for each object and assign a feature pyramid network (FPN) to each object. When performing computer vision tasks such as object detection and image classification, it is necessary to consider the size and shape of objects at different scales. This is because the same type of object may appear at different sizes and scales in an image. Therefore, the model must be able to recognize and localize objects at different scales, not just objects at a specific scale.

[0113] The Feature Pyramid Network (FPN) solves this problem by constructing multiple feature maps of different resolutions. It uses a convolutional neural network (CNN) with shared weights to extract features from the input image. Then, for each different scale, it resizes the original image by increasing or decreasing the stride. At each scale, the FPN generates a set of feature maps from high to low, which are called "pyramids". The bottom layer is the feature map obtained by upsampling the original image, and the top layer is a coarse feature map obtained by applying pooling and convolution operations.

[0114] These feature maps are then concatenated to produce higher-level feature representations. This concatenation process enables the model to capture target information at different scales and is richer in depth, thereby improving model performance. For each location, four values ​​are predicted: two offsets of the top left corner of the grid and the height and width of the predicted box (in pixels). In order to more accurately detect object types, the multi-scale features obtained from different network layers by the above FPN are fused. The network consists of two subnetwork modules. The first is a top-down architecture (the bottom-up network starts from the original image and gradually reduces the image size through multiple convolutional layers and pooling layers to obtain a series of feature maps from coarse to fine. These feature maps gradually increase in the number of channels, capturing higher and higher-level features. The bottom-up network is mainly responsible for extracting low-level features, such as edges and textures, etc.), and the second is a bottom-up architecture (the top-down network starts from the top of the pyramid and gradually restores the size of the feature map through upsampling or deconvolution operations, while adding lateral connections at each layer. The top-down network and the corresponding layers of the bottom-up network are horizontally connected so that the top-down network can obtain high-level feature information from the bottom-up network. The top-down network is mainly responsible for extracting high-level features, such as semantic information and the overall shape of objects). Each subnetwork uses deep residual network (ResNet) blocks to build a deep network. These residual blocks contain jump connections across layers. Skip connections allow layers to be skipped between input and output, allowing the network to capture higher-level features and making it easier to train. They also pass gradients back to earlier layers, alleviating the vanishing gradient problem. The RGB image is fed into the bottom-up subnetwork, and the feature map is then passed to the next subnetwork. In the top-down subnetwork, each feature map at each layer is combined to predict the object category and the corresponding candidate bounding box. This can then be trained based on the target recognition model framework by collecting a large number of images of the target object to obtain a deep learning recognition model.

[0115] Continuing with the above example, the camera in the user's AR glasses can capture the image (including RGB image and depth image) corresponding to the augmented reality interaction scene seen from the user's perspective. At this time, the augmented reality device can determine the image data corresponding to the image. The image data may include the image data of the RGB image captured by the camera when the user performs the "multi-person collaborative cup lid twisting" service and the depth data of the depth image matching the image. In addition, the augmented reality device can identify the position of the target object cup lid in the pixel coordinate system of the image as the initial position based on the preset recognition model. Afterwards, based on the initial position, the actual position of the target object in the augmented reality interaction scene where the user is located can be determined by performing a series of spatial coordinate transformations based on the RGB image and depth image corresponding to the image and the parameters of the internal and external matrices of the camera installed in the device.

[0116] In addition, when it is detected that the user needs to add a new service to the augmented display device, for example, when it is detected that the user wants to use the voice recognition service in the augmented display device, but the augmented display device does not have the voice recognition service, then the service that the user needs to add is treated as a service to be expanded, and the service to be expanded is instantiated. After that, the instantiated service is called into the preset IOC (Inversion of Control) container, and the behavior of the service to be expanded in the interactive interface is managed, and the instantiated service to be expanded is called to update the executable service in the augmented reality device.

[0117] FIG3 is a schematic diagram of a process for creating a service to be expanded for an augmented reality device provided in the present disclosure;

[0118] As shown in Figure 3, when the user touches a button in the display interface of the enhanced display device, the button is used to indicate that a new business A and a new business B are to be added to the enhanced display device. Business A and business B are the businesses to be expanded. The element data corresponding to the button is activated, and the preset interface of the control component in the enhanced display device will respond to the changes in the preset element data.

[0119] Afterwards, the business context component used to manage different types of businesses can obtain the business to be expanded determined by the user through the control component after IOC dependency injection, and the business context component can derive the business context component of business A and the business context component of business B based on the business to be expanded determined by the above user.

[0120] The business instance component used for specific business instantiation can rely on the business context component of the above-mentioned business A and the business context component of business B, derive the business instance component of business A and the business instance component of business B through the business component, and determine the tasks A1 to be instantiated, A2 to be instantiated, and A3 to be instantiated contained in business A, and determine the tasks B1 to be instantiated and B2 to be instantiated contained in business B. Afterwards, all the above-mentioned tasks to be instantiated can be injected into the IOC container using dependency injection, and all the above-mentioned tasks to be instantiated can be instantiated in the IOC container to realize the instantiation of the above-mentioned business to be expanded, that is, at this time, the IOC container will contain the instantiated business A and business B, and the above-mentioned instantiation can be the specific implementation of the functions corresponding to the above-mentioned tasks. Afterwards, the preset first control component will call the business A and business B from the preset IOC container to realize the update of the executable business in the augmented reality device.

[0121] S204: Sending the posture and the synchronization instruction corresponding to the interactive action performed by the user to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interaction scene presented by the augmented reality device according to the posture and the synchronization instruction.

[0122] When users use augmented reality devices to perform business, they may be involved in information interaction with other users, that is, multiple users may perform operations on the same target object or virtual interface object, or users may perform operations or other interactive operations on virtual interface objects created by each other.

[0123] At this time, in order to ensure the user experience of the augmented reality device, so that the interactive interface of each user's augmented reality device can promptly display the effects of these operations to all the users who perform the business together, it is necessary for the augmented reality device to share the effects of these operations in real time among each user.

[0124] For example, when there are 5 users, each holding an augmented reality device, in the augmented reality interaction scene of each augmented reality device, the online and offline status of each user can be represented by the user's virtual character image in the augmented reality interaction scene, that is, if the user is online, then the user's virtual character image will appear in the augmented reality interaction scene, otherwise, the user's virtual character image will not appear in the augmented reality interaction scene.

[0125] Then, when all five users are online and the first user is offline, that is, the first user performs an offline interactive operation through the augmented reality device held by the user, at this time, the virtual character of the first user needs to disappear, which is the synchronization instruction mentioned above that corresponds to the operation and needs to be sent synchronously to other augmented reality devices. After the synchronization is completed, the virtual character of the first user in each user's augmented reality device will disappear.

[0126] For another example, when there are five users, each holding an augmented reality device, and they all enter a cooperative pitching game, each of the five users has a virtual ball, and the five users take turns throwing the virtual ball into a box in front of them in reality. If the first user performs an interactive action of throwing a virtual ball into the box, then the augmented reality device of the first user will send the posture of the virtual ball and the synchronization instruction corresponding to the interactive action to other augmented reality devices, so that the other augmented reality devices can synchronize the augmented reality interaction scene presented by the augmented reality device according to the posture and synchronization instruction.

[0127] In addition, since the augmented reality device may also respond to the data update instruction sent by the server to update the scene, button or other components of the display interface of the augmented reality device, the synchronization instruction can be an instruction to update the above content. For example, when the server needs to simulate the scene of the sun rising and setting in nature in the display interface of the augmented reality device, once the augmented reality device receives the synchronization instruction to update the position, color and other information of the sun in the above display interface, it can respond to the synchronization instruction and update the position, color and other information of the sun in the above display interface. The sun here can be the specified object that needs to be updated mentioned above.

[0128] The process of information synchronization and information interaction between the aforementioned augmented reality device and other augmented reality devices or between the augmented reality device and the server can be specifically shown in FIG4 .

[0129] FIG4 is a schematic diagram of a flow chart of information interaction provided in the present disclosure.

[0130] As shown in Figure 4, the collaborative component in Figure 4 is responsible for managing servers, clients, and multiple network connections. The server is the intermediate node between multiple clients in information exchange, responsible for managing client additions, deletions, behavioral responses, data transmission, and other behaviors. The server here can be either an augmented display device or a real-world server. The client is an augmented display device that manages the current user's behavior transmission and the synchronization of other synchronization instruction data locally on the client. The "network connection" management component in Figure 4 contains two derived components: the network connection server management component (corresponding to the network connection with the server) and the network connection server client management component (corresponding to the network connection with the client). During the information exchange process, each role needs to be uniquely marked to facilitate differentiation during data synchronization. Therefore, a collaborative identity component is required to determine a unique identity for each synchronization instruction. In addition, the behavior management component is managed by the above-mentioned server and client, and is associated with the "network connection" management component. The behavior management component is responsible for the behavior management of all roles. The behavior management component is managed by the collaborative identity identification component. The behavior management component can clearly determine the sender and receiver of the synchronization instruction based on the information in the collaborative identity identification component to achieve stable information interaction. The transformation component is used for interface management of network communication to realize information interaction and partial information connection for subsequent data transmission.

[0131] The above-mentioned process of real-time sharing of interactive data of augmented reality devices can be specifically shown in FIG5 .

[0132] FIG5 is a flow chart of real-time sharing of interactive data of an augmented reality device provided in the present disclosure.

[0133] The client in Figure 5 refers to the augmented reality device owned by the user, and the server refers to the server or an augmented reality device designated as the server. The client manages the current user's behavior transmission and the local synchronization of collaborative information data from other clients. The server is the intermediate node between multiple clients in multi-person collaboration, responsible for managing client additions, deletions, behavior responses, data transmission, and other behaviors. Other clients in Figure 5 refer to augmented reality devices owned by users other than the user that have established an information connection with the server. The collaboration component is used to manage both the server and the clients.

[0134] The client held by the user can send a connection request carrying a creation client object to the server. The client object here can be used to represent the virtual interface object that the user wants to display in the display interface of the augmented reality device corresponding to the client.

[0135] Once the server receives the data sharing request, it can trigger the instruction to create the client object in the server through a preset callback function. The cooperative component can create the client object in the augmented reality interaction scene on the server according to the callback function, and mark the created client object with the client.

[0136] Afterwards, the server can mark the client as information that requires collaboration (i.e., synchronization variables) and serialize the synchronization variables (i.e., convert the synchronization variables into binary data). The server can send the information of the client object carrying the serialized synchronization variables to other clients through the collaboration component.

[0137] At the same time, the collaboration component can send the client object and the client's tag (i.e., the subject affiliation information in Figure 5) to the client. After that, the client can grant the server the permission to create the client object in the client's augmented reality scene and present the client object in the client's augmented reality scene.

[0138] After receiving the information of the client object carrying the serialized synchronization variables, other clients can parse the information and deserialize the information of the client object carrying the serialized synchronization variables (that is, convert the binary data into the form of synchronization variables) to obtain the information of the client and the client object created by the client, and synchronously create the client object in the augmented reality scene created by other clients.

[0139] If the user performs an operation on the client object in the client, then similarly, through the above steps, other clients can also receive the operation performed by the user on the client object and synchronously update the status of the client in the augmented reality interaction scene of other clients to achieve the purpose of the above-mentioned synchronous sharing.

[0140] Continuing with the "multi-person collaboration to twist the cup lid" service in the above example, when users execute the "multi-person collaboration to twist the cup lid" in the above example, it may involve multiple users performing some operations on the target object cup lid in turn, then the augmented reality device can obtain the synchronization instructions issued by each user for the cup lid in turn. For example, there are 3 users (user A, user B, user C) participating in the game. User A first issues a synchronization instruction to move the cup lid 3 cm to the left. At this time, the augmented reality device needs to send the synchronization instruction to other augmented reality devices, so that other augmented reality devices can synchronize the augmented reality interaction scenes presented by these other augmented reality devices according to the posture of the cup lid after user A moves it. That is, after user A moves the cup lid 3 cm to the left, after the above synchronization, the posture of the cup lid recognized by the augmented reality devices of users B and C is also moved 3 cm to the left based on the original cup lid posture.

[0141] During the synchronization process, the AR device can synchronize scene data with other users' AR devices based on a pre-set data transmission strategy and method. The data transmission strategy can also be user-specified. For example, a user can specify that the AR device transmit video at a speed of 30 frames per second.

[0142] The above-mentioned specific data transmission process can be shown in FIG6 , which is a schematic diagram of a flow chart of data transmission by an augmented reality device provided in the present disclosure.

[0143] As can be seen from Figure 6, the step of data transmission of augmented reality devices is completed through the hybrid network communication module. The hybrid network communication module is mainly composed of five logical layers: physical layer, transport layer, policy layer, data layer, and multi-channel communication control.

[0144] The data layer contains data that needs to be communicated, such as application basic data, interaction data, status data, scene data, virtual-reality fusion data, collection data, analysis and evaluation data, and other augmented reality application data. The data layer is mainly used to classify and label the data that needs to be communicated. The labels are divided into data strategy S and data transmission method T.

[0145] The strategy layer is divided into three types: high-frequency real-time transmission strategy, low-frequency online transmission strategy, and signal-triggered transmission strategy. The high-frequency real-time transmission strategy is aimed at communication needs with high real-time requirements for data transmission. The transmission frequency adopts a predefined unified frequency, and the amount of data transmitted is limited and optimized based on the specific network conditions. The low-frequency online transmission strategy is aimed at scenarios with low data update frequency requirements. It can transmit large amounts of data and adopts a predefined unified frequency for transmission. The signal-triggered transmission strategy is aimed at user business needs and can be triggered by internal processes or external inputs. There is no specific limit on the amount of data transmitted.

[0146] The transport layer utilizes three transmission methods: lightweight communication, asynchronous high-capacity communication, and streaming data communication. Lightweight communication is suitable for small data volumes and is implemented using basic socket communication. Streaming data communication is suitable for real-time audio and video communication, which requires high real-time performance and is typically implemented using Web Real-Time Communication (WebRTC). Asynchronous high-capacity communication is suitable for large data volumes and requires high transmission speed and quality.

[0147] The multi-channel communication control layer is used to judge, integrate, make decisions, and verify the strategies and specific methods for data transmission. It mainly includes the construction of a converged network (when business needs require the integration of multiple data strategies or transmission methods), protocol adaptation (protocol compatibility of different data transmission interfaces), data classification and tagging, data serialization and deserialization, communication session construction (for example, building a chat dialog box), concurrent thread management to improve data transmission efficiency and stability, and data information verification (to verify the security of transmitted data).

[0148] It can be seen from the above method that by deploying a plurality of modules that can collaborate with each other for the augmented reality device, the user using the augmented reality device can use the augmented reality device to perform the interaction between virtual and reality, and the operations performed by the user when performing the interactive service can be synchronized to other users, so that the user can share the interaction between virtual and reality with other users.

[0149] The above is the augmented reality device and one or more implementation methods of the present disclosure. Based on the same idea, the present disclosure also provides a corresponding augmented reality apparatus, as shown in FIG7 .

[0150] FIG7 is a schematic diagram of an augmented reality device provided by the present disclosure, comprising:

[0151] A creation module 701 is configured to create an augmented reality interaction scene based on an augmented reality interaction scene creation request received from a user;

[0152] Binding module 702 is configured to bind, for each interface element, the element variable of the interface element to the data variable of the element data required for rendering the interface element, based on the element variables preset for the interface element and the data variables preset for the element data, thereby binding the interface element to the element data corresponding to the interface element, so that the interface element affected by the user's interactive action and the element data required for rendering the interface element are bound in the augmented reality interaction scene through the defined designated object class, wherein the designated object class includes the object class corresponding to the interface element and the object class corresponding to the interface element data;

[0153] The determination and display module 703 is used to, when monitoring the interactive action performed by the user, determine the augmented reality interaction scene that the user wants to display in the augmented reality device, call the business scene data required in the augmented reality interaction scene and the common data used in different augmented reality interaction scenes, so as to display the virtual interface object that the user wants to display in the display interface of the augmented reality device, and determine the position and posture of the target object in the augmented reality interaction scene where the user is located;

[0154] The synchronization module 704 is used to send the synchronization instructions corresponding to the posture and the interactive action performed by the user to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interaction scene presented by the augmented reality device according to the posture and the synchronization instructions.

[0155] Optionally, the device further comprises:

[0156] The updating module 705 is used to update the elements in the display interface of the augmented reality device in real time according to the update status of the data bound to the interface elements in the backend.

[0157] Optionally, the determination and display module 703 is used to collect image data of the augmented reality scene; identify the posture of the target object in the image represented by the image data as an initial posture according to a preset recognition model; and determine the posture of the target object in the augmented reality interaction scene where the user is located based on the initial posture and the image data.

[0158] Optionally, the synchronization module 704 is used to determine scene data corresponding to the posture and the synchronization instruction corresponding to the interactive action performed by the user in the augmented reality interaction scene; and transmit the scene data to other augmented reality devices.

[0159] Optionally, the synchronization module 704 is further configured to obtain a data transmission strategy determined by a user; and transmit the scene data to other augmented reality devices according to the data transmission strategy determined by the user.

[0160] Optionally, the device further comprises:

[0161] The business expansion module 706 is used to obtain the new business demand instruction sent by the user; in response to the new business demand instruction, determine the business to be expanded, instantiate the business to be expanded through a specified container, and update the executable business in the augmented reality device according to the instantiated business to be expanded.

[0162] The present disclosure also provides a computer-readable storage medium, which stores a computer program. The computer program can be used to execute the augmented reality method provided in FIG. 1 .

[0163] This disclosure also provides a schematic structural diagram of an electronic device corresponding to Figure 1, shown in Figure 8. As shown in Figure 8, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile storage into the memory and then runs it to implement the augmented reality method described in Figure 2 above.

[0164] Of course, in addition to software implementation, this disclosure does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0165] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0166] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0167] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0168] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0169] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0171] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0173] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0174] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0175] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0176] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0177] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] The present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0179] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0180] The foregoing is merely an embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure are intended to be included within the scope of the claims of the present disclosure.

Claims

1. An augmented reality device, characterized in that: The device is provided with a data-driven interactive interface module, a universal and extensible business module, an identification and tracking module, a multi-person collaboration module, and a hybrid network communication module; The data-driven interactive interface module is used to instantiate, for each interface element included in the interactive interface for interacting with the user, the class corresponding to the interface element to obtain a first instantiated object, and instantiate the element data class pre-bound to the interface element to obtain a second instantiated object, and for each specified variable included in the first instantiated object, bind the specified variable to a specified variable with the same name as the specified variable among the specified variables included in the second instantiated object, so that when the value of any specified variable bound to the first instantiated object in the second instantiated object changes, the presentation of the interface element corresponding to the specified variable bound to the any specified variable in the first instantiated object in the interactive interface changes synchronously; The universal extensible service module is used to pre-save the container corresponding to the service executed by the augmented reality device, and when executing the target service, call the component in the container corresponding to the target service to execute the target service based on the called component; create a new service to be expanded through the preset components, and manage the creation status of the service to be expanded; The recognition and tracking module is used to perform image recognition on the image data collected by the augmented reality device to determine the position and posture of the target object in the augmented reality interaction scene where the user is located; The multi-person collaboration module is used to respond to a data synchronization instruction and send a data synchronization instruction generated by a local device to other augmented reality devices, wherein the data synchronization instruction is used to control a target object in the augmented reality interaction scene or update the state of the target object; The hybrid network communication module is used to determine a transmission strategy according to different types of data to be transmitted generated in the augmented reality interaction scene, and transmit different types of data to be transmitted according to different transmission strategies.

2. The augmented reality device according to claim 1, wherein: The data-driven interactive interface module is also used to update the elements in the interface in real time according to the update status of the data bound to the interface elements in the display interface of the augmented reality device at the back end.

3. An augmented reality method, characterized in that: The method is applied to the augmented reality device according to claim 1 or 2, and the method comprises: Creating an augmented reality interaction scene based on an augmented reality interaction scene creation request received from a user; According to the element variables preset for the interface elements and the data variables preset for the element data, for each interface element, the element variables of the interface element are bound to the data variables of the element data required for rendering the interface element, so as to realize the binding of the interface element with the element data corresponding to the interface element, so that the interface elements affected by the user's interactive actions and the element data required for rendering the interface elements are bound in the augmented reality interactive scene through the defined designated object class, wherein the designated object class includes the object class corresponding to the interface element and the object class corresponding to the interface element data; When an interactive action performed by the user is monitored, the augmented reality interaction scene to be displayed by the user in the augmented reality device is determined, and the business scene data required in the augmented reality interaction scene and the common data used in different augmented reality interaction scenes are called to display the virtual interface object to be displayed by the user in the display interface of the augmented reality device, and the position and posture of the target object in the augmented reality interaction scene where the user is located are determined; The posture and the synchronization instruction corresponding to the interactive action performed by the user are sent to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interaction scene presented by the augmented reality device according to the posture and the synchronization instruction.

4. The method according to claim 3, characterized in that The method further includes: updating the elements in the interface in real time according to the update status of the data bound to the interface elements in the display interface of the augmented reality device at the back end.

5. The method according to claim 3, characterized in that Determining the position and posture of a target object in an augmented reality interaction scene where the user is located includes: Collecting image data of the augmented reality scene; Identify the position and posture of the target object in the image represented by the image data according to a preset recognition model is the initial posture; The posture of a target object in the augmented reality interaction scene where the user is located is determined according to the initial posture and the image data.

6. The method according to claim 3, characterized in that Sending the posture and the synchronization instruction corresponding to the interactive action performed by the user to other augmented reality devices, including: Determine the scene data corresponding to the posture and the synchronization instruction corresponding to the interactive action performed by the user in the augmented reality interaction scene; The scene data is transmitted to other augmented reality devices.

7. The method according to claim 6, characterized in that The scene data is transmitted to other augmented reality devices, and the method further comprises: Obtaining the data transmission strategy determined by the user; The scene data is transmitted to other augmented reality devices according to the data transmission strategy determined by the user.

8. The method according to claim 3, characterized in that The method further comprises: Obtain new business demand instructions sent by users; In response to the newly added service requirement instruction, the service to be expanded is determined, the service to be expanded is instantiated through a specified container, and the executable service in the augmented reality device is updated according to the instantiated service to be expanded.

9. An augmented reality device, characterized in that: The device is used to implement an augmented reality method based on the augmented reality device according to claim 1 or 2, comprising: A creation module, used to create an augmented reality interaction scene based on an augmented reality interaction scene creation request received from a user; A binding module, for binding the element variable of each interface element with the data variable of the element data required for rendering the interface element according to the element variable preset for the interface element and the data variable preset for the element data, so as to realize the binding of the interface element with the element data corresponding to the interface element, so that the interface element affected by the user's interactive action and the element data required for rendering the interface element are bound in the augmented reality interactive scene through the defined designated object class, wherein the designated object class includes the object class corresponding to the interface element and the object class corresponding to the interface element data; A determination and display module, which is used to determine the augmented reality interaction scene to be displayed by the user in the augmented reality device when an interactive action performed by the user is monitored, and to call the business scene data required in the augmented reality interaction scene and the common data used in different augmented reality interaction scenes, so as to display the virtual interface object to be displayed by the user in the display interface of the augmented reality device, and determine the position and posture of the target object in the augmented reality interaction scene where the user is located; A synchronization module is used to send synchronization instructions corresponding to the posture and the interactive action performed by the user to other augmented reality devices, so that the other augmented reality devices synchronize the augmented reality interaction scene presented by the augmented reality device according to the posture and the synchronization instructions.

10. The device according to claim 9, characterized in that The device also includes: The updating module is used to update the elements in the display interface of the augmented reality device in real time according to the update status of the data bound to the interface elements in the back end.

11. The device according to claim 9, characterized in that The determination and display module is used to collect image data of the augmented reality scene; identify the position of the target object in the image represented by the image data as an initial position according to a preset recognition model; The posture of a target object in the augmented reality interaction scene where the user is located is determined according to the initial posture and the image data.

12. The device according to claim 9, characterized in that The synchronization module is used to determine the scene data corresponding to the synchronization instruction corresponding to the posture and the interactive action performed by the user in the augmented reality interaction scene; and transmit the scene data to other augmented reality devices.

13. The device according to claim 12, characterized in that The synchronization module is also used to obtain a data transmission strategy determined by a user; and transmit the scene data to other augmented reality devices according to the data transmission strategy determined by the user.

14. The device according to claim 9, characterized in that The device also includes: The business expansion module is used to obtain the new business demand instruction sent by the user; in response to the new business demand instruction, determine the business to be expanded, instantiate the business to be expanded through the specified container, and update the executable business in the augmented reality device according to the instantiated business to be expanded.

15. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 3 to 8 is implemented.

Images