Interactive display method, device, equipment, medium and product based on augmented reality

Abstract

The present application discloses an augmented reality-based interactive display method, device, equipment, medium and product, belonging to the field of augmented reality technology. The method comprises: constructing a three-dimensional model and a point cloud map of a real scene in the cloud, and determining a corresponding relationship; obtaining at least one of the following: a first interactive content and a first posture information added to the three-dimensional model, and a second interactive content and a second posture information added to the point cloud map; when the first interactive content and the first posture information are included, sending the first interactive content and the first posture information to a third user located in the real scene, adding the first interactive content to a corresponding position of the point cloud map, and displaying the first interactive content to the third user; when the second interactive content and the second posture information are included, adding the second interactive content to a corresponding position of the three-dimensional model, and sending the three-dimensional model after adding the second interactive content to a fourth user who is not in the real scene.

Description

Technical Field

[0001] The present application belongs to the field of augmented reality technology, and specifically relates to an augmented reality-based interactive display method, device, equipment, medium and product. Background Art

[0002] At present, augmented reality (AR) technology can simultaneously display real scenes and pre-created virtual content on the client side based on users scanning real scenes. In the relevant technical field, most AR devices are usually in a stand-alone state or can only transmit data with the cloud. Through the user's scanning of the real scene, the corresponding virtual content can be presented on the user's AR device. However, for remote users who are not in the real scene, it is difficult to interact with users in the real scene. It can be seen that there is a problem of poor AR interaction effect between real scene users and remote users in the relevant technical field. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an augmented reality-based interactive display method, device, equipment, medium and product, which can solve the problem of poor AR interaction effect between real-scene users and remote users in the relevant technical field.

[0004] In a first aspect, an embodiment of the present application provides an interactive display method based on augmented reality, comprising:

[0005] The cloud constructs a three-dimensional model and a point cloud map of the real scene based on image data uploaded by a user in the real scene, and determines a corresponding relationship between the three-dimensional model and the point cloud map;

[0006] The cloud obtains interactive content information, wherein the interactive content information includes at least one of the following: first interactive content and first posture information added to the three-dimensional model by a first user who is not in the real scene, and second interactive content and second posture information added to the point cloud map by a second user who is in the real scene;

[0007] In the case where the interactive content information includes the first interactive content and the first posture information, the cloud sends the first interactive content and the first posture information to a third user located in the real scene based on the corresponding relationship, determines the relative posture information of the third user and the first interactive content based on the first posture information, and adds the first interactive content to a corresponding position in the field of view of the third user based on the relative posture information;

[0008] In the case where the interactive content information includes the second interactive content and the second posture information, the cloud adds the second interactive content at the corresponding position of the three-dimensional model based on the corresponding relationship, the second interactive content and the second posture information, and sends the three-dimensional model with the second interactive content added to a fourth user who is not in the real scene.

[0009] Optionally, the correspondence between the three-dimensional model and the point cloud map includes a correspondence established in the following manner:

[0010] Determining points at multiple spatial locations in the point cloud map;

[0011] Determining points at a plurality of spatial locations in the three-dimensional model;

[0012] Match the points at multiple spatial positions in the point cloud map with the points at multiple spatial positions in the three-dimensional model, and generate a correspondence between the three-dimensional model and the point cloud map based on the matching results, wherein the matching results represent the points in the point cloud map and the three-dimensional model that correspond to the same spatial position.

[0013] Optionally, before the cloud obtains the interactive content information, the method further includes at least one of the following:

[0014] The cloud sends the three-dimensional model to the first user, and enables the first user to edit the interactive content of the three-dimensional model;

[0015] The cloud sends the point cloud map to the second user, and enables the second user to edit the interactive content of the point cloud map.

[0016] Optionally, determining a relative posture relationship between the third user and the first interactive content based on the first posture information, and adding the first interactive content at a corresponding position in the field of view of the third user based on the relative posture relationship includes:

[0017] The cloud obtains third position information of the third user in the point cloud map based on the VPS technology;

[0018] The cloud determines, based on the third posture information and the first posture information, relative posture information of the first interactive content with the third posture information as a reference;

[0019] According to the relative posture information, the first interactive content is displayed at a relative position of the third user's field of view.

[0020] Optionally, the cloud adds the second interactive content at a corresponding position of the three-dimensional model based on the corresponding relationship, the second interactive content and the second posture information, including:

[0021] The cloud determines, based on the corresponding relationship, that the second posture information corresponds to the fourth posture information in the three-dimensional model; wherein the corresponding relationship includes a one-to-one correspondence between the point positions in the point cloud map and the point positions in the three-dimensional model;

[0022] The cloud adds the second interactive content at a position corresponding to the fourth posture information in the three-dimensional model.

[0023] Optionally, the first interactive content or the second interactive content includes at least one of the following: image, text, audio, video, or three-dimensional model.

[0024] In a second aspect, an embodiment of the present application provides an interactive display device based on augmented reality, comprising:

[0025] A first acquisition module is used to construct a three-dimensional model and a point cloud map of the real scene in the cloud based on image data uploaded by a user in the real scene, and to determine a corresponding relationship between the three-dimensional model and the point cloud map;

[0026] A second acquisition module is used to acquire interactive content information in the cloud, wherein the interactive content information includes at least one of the following: first interactive content and first posture information added to the three-dimensional model by a first user who is not in the real scene, and second interactive content and second posture information added to the point cloud map by a second user who is in the real scene;

[0027] A first processing module, configured to, when the interactive content information includes the first interactive content and the first posture information, send the first interactive content and the first posture information to a third user located in the real scene based on the corresponding relationship, determine the relative posture information between the third user and the first interactive content based on the first posture information, and add the first interactive content to a corresponding position in the field of view of the third user based on the relative posture information;

[0028] A second processing module is used for adding the second interactive content to the corresponding position of the three-dimensional model based on the corresponding relationship, the second interactive content and the second posture information in the cloud when the interactive content information includes the second interactive content and the second posture information, and sending the three-dimensional model with the second interactive content added to a fourth user who is not in the real scene.

[0029] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the augmented reality-based interactive display method as described in the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the interactive display method based on augmented reality as described in the first aspect are implemented.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the augmented reality-based interactive display method as described in the first aspect.

[0032] The interactive display method based on augmented reality in the embodiment of the present application displays the real scene and virtual content to the users in the real scene through the point cloud map, and can also display the real scene and virtual content to the users who are not in the real scene through the three-dimensional model. Through the correspondence between the point cloud map and the three-dimensional model, when the user in the real scene adds interactive content to the point cloud map, the interactive content is added to the corresponding position of the three-dimensional model, and when the user who is not in the real scene adds interactive content to the three-dimensional model, the interactive content is added to the corresponding position of the point cloud map, so that the users in the real scene and the users who are not in the real scene can share the same real scene and related virtual content and newly added interactive content. In this way, the embodiment of the present application meets the requirements of cross-space communication and interaction between multiple users, and the interactive content of any party can be fed back to the other party in a timely and effective manner, so that the interactive information transmission is timely and efficient, the connectivity between users is enhanced, and the interactive effect between real scene users and remote users is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 This is one of the flow charts of an interactive display method based on augmented reality provided in an embodiment of the present application;

[0035] Figure 2 This is a second flow chart of an interactive display method based on augmented reality provided in an embodiment of the present application;

[0036] Figure 3 This is a third flow chart of an interactive display method based on augmented reality provided in an embodiment of the present application;

[0037] Figure 4 is a structural schematic diagram of an interactive display device based on augmented reality provided in an embodiment of the present application;

[0038] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0040] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0041] The following, in conjunction with the accompanying drawings, describes in detail the augmented reality-based interactive display method, device and related equipment provided by the embodiments of the present application through specific embodiments and their application scenarios.

[0042] Please refer to Figure 1 , Figure 1 This is one of the flowcharts of an interactive display method based on augmented reality provided in the embodiment of the present application. Figure 1 As shown, the interactive display method based on augmented reality specifically includes the following steps:

[0043] Step 101: The cloud constructs a three-dimensional model and a point cloud map of the real scene based on image data uploaded by a user in the real scene, and determines the corresponding relationship between the three-dimensional model and the point cloud map.

[0044] It should be noted that the above-mentioned cloud can be a server, data processing platform, etc. set between the remote user device and the user device of the real scene, which can realize functions such as data reception, data processing, data storage, access and sharing, and provide a platform for interactive communication between the user device of the real scene and the remote user device. In addition, the user of the above-mentioned real scene can refer to the user device, user terminal, etc. located in the real scene. The user device involved in the embodiment of the present application may be a display device that provides data connectivity to the user, etc. It can be a display device of a smart phone, AR glasses, portable, pocket-sized, handheld or other computer, which is not limited in the embodiment of the present application. Among them, the user of the real scene (i.e., the user device) can scan the real scene through the data acquisition device set by itself, and the user of the real scene uploads the point cloud map and the three-dimensional model to the cloud storage, which is convenient for the cloud to obtain data at any time for data processing, and the cloud realizes the construction of the point cloud map and the three-dimensional model.

[0045] In addition, the above-mentioned point cloud map may refer to a point cloud map, which is composed of a large number of points in the physical space, and each point may contain information such as coordinates, color, and intensity. At the same time, the three-dimensional model may refer to a three-dimensional model, a three-dimensional reconstruction model, etc., which can accurately represent the shape and structure of an object in a three-dimensional space. User devices that are not in the real scene can obtain it in real time in the cloud and display the three-dimensional model in real time.

[0046] Specifically, the three-dimensional model and point cloud map of the real scene can be constructed by the user device in the real scene through the image data of the real scene. For example, the image data of the real scene can be obtained by directly scanning the real scene through the visual camera in the real scene. The visual camera can be set in the user device of the real scene, or it can be set separately and the collected image data can be sent to the user device of the real scene. For example, when constructing a three-dimensional model, features such as edges, corners, etc. can be extracted from the collected two-dimensional image data, and then a three-dimensional model can be constructed using these feature information. Specifically, algorithms such as structured light, stereo vision or visual induced ranging can be applied to obtain the depth information of the object and generate a three-dimensional model. When constructing a point cloud map, the depth information in the image data can be used to generate a point cloud for each point according to its actual position in space, forming a visual dense set of colored points.

[0047] Furthermore, the correspondence between the three-dimensional model and the point cloud map can be a correspondence in physical space position, that is, a point or object located at a certain coordinate in the three-dimensional model can correspond to a point or object located at another coordinate in the point cloud map. The coordinates of two corresponding points or objects in the three-dimensional model and the point cloud map may be different, but their physical space positions corresponding to the real scene are the same.

[0048] Based on the above settings of the three-dimensional model, the point cloud map, and the corresponding relationship between the three-dimensional model and the point cloud map, the three-dimensional model and the point cloud map can describe the same spatial scene and virtual content.

[0049] It is worth mentioning that the specific embodiment of the augmented reality-based interactive display method provided in the present application can be applied to scenarios that require real-time interaction with remote users. Users in the real scene display virtual content through VR technology, and remote users can obtain the three-dimensional models uploaded by users in the real scene in the cloud for access and display.

[0050] Step 102, adding a first interactive content to a certain point in the image, and then obtaining the first interactive content information added by the first user from the cloud, wherein the interactive content information includes at least one of the following: a first interactive content and a first posture information added to the three-dimensional model by a first user who is not in the real scene, and a second interactive content and a second posture information added to the point cloud map by a second user who is in the real scene.

[0051] It is understandable that when a first user who is not in the real scene needs to interact with a user in the real scene, the interactive content in the three-dimensional model and the posture information corresponding to the first interactive content (i.e., the first posture information) can be stored and further processed as interactive content information. When a second user in the real scene needs to interact with a remote user for the real scene, the second user in the real scene can add a second interactive content at a certain point in the point cloud map, and then the cloud obtains the second interactive content of the second user and the second posture information of the second interactive content.

[0052] The first pose information may include the position coordinates (e.g., X-axis, Y-axis, Z-axis coordinates, etc.) and orientation angles (e.g., yaw, pitch, roll, etc.) of the first interactive content in the three-dimensional model. The second pose information may also include the position coordinates and orientation angles of the second interactive content in the point cloud map. It should be understood that the pose information in the embodiment of the present application is equivalent to an anchor point, which can provide the position information of the interactive content.

[0053] Specifically, the interactive content information obtained by the cloud can include interactive content and the position information corresponding to the interactive content, wherein the interactive content can include the type of interactive content added by the user, content description, user identifier, timestamp, etc. Thus, the cloud obtains interactive content information, so that all interactive content is centrally stored, which is convenient for subsequent data processing and realizes real-time sharing of interactive content between remote users and users in real scenes.

[0054] In this way, users in the real scene and remote users can participate in interaction and collaboration at the same time, supporting multi-user interaction in different environments. Even two users with different actual physical locations (the first user who is not in the real scene and the second user who is in the real scene) can add and share interactive content. The connectivity between users effectively improves the interaction effect between real scene users and remote users.

[0055] Step 103: When the interactive content information includes the first interactive content and the first posture information, the cloud sends the first interactive content and the first posture information to a third user located in the real scene based on the corresponding relationship, determines the relative posture information of the third user and the first interactive content based on the first posture information, and adds the relative posture information at the corresponding position in the field of view of the third user based on the relative posture information.

[0056] In a specific embodiment of the present application, Figure 2 As shown, when a first user who is not in the real scene needs to interact with a third user who is in the real scene, the third user needs to participate in the interaction with the remote first user who is not in the real scene, and the first interactive content and the first position information can be sent from the cloud to the third user in the real scene. Since the first position information can describe the specific position of the first interactive content in the three-dimensional model, the third user can first determine the specific position of the first interactive content in the three-dimensional model based on the first position information, and correspond the specific position of the first interactive content to the position in the point cloud map, so as to determine the corresponding position in the point cloud map.

[0057] In addition, the first interactive content can be displayed in the point cloud map according to the perspective of the third user in the point cloud map. Therefore, the cloud can obtain the posture information of the third user in the point cloud map, and determine the relative posture information of the third user and the first interactive content based on the posture information of the third user and the first posture information. Furthermore, the first interactive content can be added to the corresponding position under the relative posture information in the point cloud map from the perspective of the third user. Finally, the first interactive content added at the corresponding position is displayed to the third user to achieve interaction with the first user.

[0058] It should be noted that in the embodiment of the present application, the user and the third user who construct the three-dimensional model and the point cloud map can be the same user (i.e., user device) or different users. That is, the embodiment of the present application can realize interaction between multiple users. After the first user who is not in the real scene adds the first interactive content through the three-dimensional model, the third user in the real scene and the user who constructs the three-dimensional model and the point cloud map can both obtain the first interactive content and the first posture information. Then, according to the first posture information, the first interactive content can be added to the corresponding position of the point cloud map. Therefore, the user who uploaded the image data and the third user can both see the added first interactive content.

[0059] In this way, the above embodiment realizes the interaction between remote users and users in the real scene, meets the requirements of cross-space communication and interaction between multiple users, makes the interactive information transmission timely and efficient, enhances the connectivity between users, and effectively improves the interactive effect between real scene users and remote users.

[0060] Step 104, when the interactive content information includes the second interactive content and the second posture information, the cloud adds the second interactive content at the corresponding position of the three-dimensional model based on the corresponding relationship, the second interactive content and the second posture information, and sends the three-dimensional model with the second interactive content added to a fourth user who is not in the real scene.

[0061] In another specific embodiment of the present application, Figure 3 As shown, when the second user in the real scene adds the second interactive content and needs to interact with the fourth user who is not in the real scene, the cloud can directly add the second interactive content at the corresponding position of the three-dimensional model according to the second posture information and the corresponding relationship. It should be noted that the corresponding relationship includes a one-to-one correspondence between the point cloud map and each point in the three-dimensional model. Therefore, after determining the second posture information describing the second interactive content in the point cloud map, the second interactive content can be added to the position corresponding to the second posture information in the three-dimensional model. Subsequently, the three-dimensional model with the second interactive content added can be sent to the fourth user, and the fourth user can display the added three-dimensional model to interact with the second user.

[0062] In addition, the fourth user in the embodiment of the present application may be the same user as the first user in the aforementioned embodiment, or may be a different user. That is, in the embodiment of the present application, both users in the real scene and users not in the real scene can add interactive content and share it with each other, thereby realizing interactive sharing between users in the real scene and remote users, and improving the interactive effect between users in the real scene and remote users.

[0063] Optionally, the correspondence between the three-dimensional model and the point cloud map includes a correspondence established in the following manner:

[0064] Determining points at multiple spatial locations in the point cloud map;

[0065] Determining points at a plurality of spatial locations in the three-dimensional model;

[0066] Match the points at multiple spatial positions in the point cloud map with the points at multiple spatial positions in the three-dimensional model, and generate a correspondence between the three-dimensional model and the point cloud map based on the matching results, wherein the matching results represent the points in the point cloud map and the three-dimensional model that correspond to the same spatial position.

[0067] In an optional embodiment of the present application, point correspondence can be achieved by matching points at multiple spatial positions in the point cloud map with points at multiple spatial positions in the three-dimensional model, thereby establishing a corresponding relationship between the three-dimensional model and the point cloud map.

[0068] For example, when the real scene is a bedroom, a 3D model and point cloud map corresponding to the bedroom are constructed. If there is a floor lamp placed in a corner of the bedroom, a corresponding floor lamp is also set in the 3D model. Similarly, a floor lamp is also set in the point cloud map. At this time, the floor lamps in the 3D model and the point cloud map correspond to the same floor lamp placed in the corner of the bedroom. For another example, when the real scene is a VR interactive corner of a park, a 3D model and point cloud map corresponding to the VR interactive corner are constructed. The VR interactive corner is set with points for residents to stand and interact. This standing interaction point will be set in both the 3D model and the point cloud map.

[0069] Therefore, each point in the point cloud map can find a corresponding point in the three-dimensional model, and the points or objects in the point cloud map correspond one to one with the points or objects in the three-dimensional model. In this way, by setting the three-dimensional model, the point cloud map, and the corresponding relationship between the three-dimensional model and the point cloud map, the three-dimensional model and the point cloud map can describe the same spatial scene and virtual content, and when the point cloud map interacts, it can correspond to the same spatial position in the three-dimensional model, and when the three-dimensional model interacts, it can also correspond to the same spatial position in the point cloud map, so as to realize the interactive sharing between users in the real scene and remote users, and improve the interactive effect between users in the real scene and remote users.

[0070] Optionally, before the cloud obtains the interactive content information, the method further includes at least one of the following:

[0071] The cloud sends the three-dimensional model to the first user, and enables the first user to edit the interactive content of the three-dimensional model;

[0072] The cloud sends the point cloud map to the second user, and enables the second user to edit the interactive content of the point cloud map.

[0073] In another optional embodiment of the present application, Figure 2 and Figure 3 As shown, the cloud can have clear user role definition and editing permission configuration functions to control different users' access and operation permissions to the 3D model and point cloud map. Specifically, before the first user needs to add the first interactive content, the cloud can open the interactive content editing permission of the 3D model for the first user. Before the second user needs to add the second interactive content, the cloud can open the interactive content editing permission of the point cloud map for the second user.

[0074] In this way, by opening up interactive content editing permissions for users, an efficient, flexible and collaborative environment is established in the augmented reality interaction process, which promotes interaction between users, improves the efficiency of interactive content creation and management, effectively reduces the risk of erroneous editing, ensures data security in the interactive process, and enhances user experience.

[0075] Optionally, determining a relative posture relationship between the third user and the first interactive content based on the first posture information, and adding the first interactive content at a corresponding position in the field of view of the third user based on the relative posture relationship includes:

[0076] The cloud obtains third position information of the third user in the point cloud map based on the VPS technology;

[0077] The cloud determines, based on the third posture information and the first posture information, relative posture information of the first interactive content with the third posture information as a reference;

[0078] According to the relative posture information, the first interactive content is displayed at a relative position of the third user's field of view.

[0079] In the specific embodiments of the present application, Figure 2As shown, as a third user in the real scene, the point cloud map of the real scene can be obtained from the cloud, and through the Visual Positioning System (VPS) visual positioning system technology, the third user can obtain its own third posture information in the current point cloud map. Subsequently, according to the first interactive content and the first posture information sent by the cloud, the third posture information of the user can be used as a reference to determine the relative posture information corresponding to the first posture information in the point cloud map. In this way, from the perspective of the third user, the first interactive content can be added at the relative posture information in the point cloud map, so that the field of view with the first interactive content added can be displayed to the third user in real time.

[0080] It can be seen that the embodiments of the present application can synchronously display interactive content in real time between users in the real scene and remote users, realize the integrated display of the real scene and interactive content according to the third user in the real scene, and the relative posture of the first interactive content remains consistent under the display of the third user, thereby improving the interactive effect between users in the real scene and remote users.

[0081] Optionally, the cloud adds the second interactive content at a corresponding position of the three-dimensional model based on the corresponding relationship, the second interactive content and the second posture information, including:

[0082] The cloud determines, based on the corresponding relationship, that the second posture information corresponds to the fourth posture information in the three-dimensional model; wherein the corresponding relationship includes a one-to-one correspondence between the point positions in the point cloud map and the point positions in the three-dimensional model;

[0083] The cloud adds the second interactive content at a position corresponding to the fourth posture information in the three-dimensional model.

[0084] In another specific embodiment of the present application, Figure 3 As shown, since the second posture information can describe the specific position of the second interactive content in the point cloud map, the cloud can first determine the specific position of the second interactive content in the point cloud map according to the second posture information, and correspond the specific position of the second interactive content to the position in the three-dimensional model, so as to determine the fourth posture information in the three-dimensional model. Furthermore, the second interactive content can be added to the fourth posture information in the three-dimensional model. Finally, the remote user displays the three-dimensional model after adding the second interactive content, which improves the interaction effect between the real scene user and the remote user.

[0085] Optionally, the first interactive content or the second interactive content includes at least one of the following: image, text, audio, video, or three-dimensional model.

[0086] In some optional embodiments, the types of the first interactive content and the second interactive content may be various, and the present application does not limit the types of interactive content, which may specifically include images, text, audio, video, three-dimensional models, etc. Among them, three-dimensional models can be embedded in three-dimensional models to obtain a more immersive experience, such as displaying detailed components or functions in education and training scenarios.

[0087] Please refer to Figure 4 , Figure 4 : is a structural diagram of an interactive display device based on augmented reality provided in an embodiment of the present application, which specifically includes:

[0088] A first acquisition module 401 is used to construct a three-dimensional model and a point cloud map of the real scene in the cloud based on image data uploaded by a user in the real scene, and to determine a corresponding relationship between the three-dimensional model and the point cloud map;

[0089] A second acquisition module 402 is configured to acquire interactive content information in the cloud, wherein the interactive content information includes at least one of the following: first interactive content and first pose information added to the three-dimensional model by a first user who is not in the real scene, and second interactive content and second pose information added to the point cloud map by a second user who is in the real scene;

[0090] A first processing module 403 is configured to, when the interactive content information includes the first interactive content and the first posture information, send the first interactive content and the first posture information to a third user located in the real scene based on the corresponding relationship, determine the relative posture information between the third user and the first interactive content based on the first posture information, and add the first interactive content to a corresponding position in the field of view of the third user based on the relative posture information;

[0091] The second processing module 404 is used for adding the second interactive content to the corresponding position of the three-dimensional model based on the corresponding relationship, the second interactive content and the second posture information in the cloud when the interactive content information includes the second interactive content and the second posture information, and sending the three-dimensional model with the second interactive content added to a fourth user who is not in the real scene.

[0092] Optionally, the correspondence between the three-dimensional model and the point cloud map includes a correspondence established in the following manner:

[0093] Determining points at multiple spatial locations in the point cloud map;

[0094] Determining points at a plurality of spatial locations in the three-dimensional model;

[0095] Match the points at multiple spatial positions in the point cloud map with the points at multiple spatial positions in the three-dimensional model, and generate a correspondence between the three-dimensional model and the point cloud map based on the matching results, wherein the matching results represent the points in the point cloud map and the three-dimensional model that correspond to the same spatial position.

[0096] Optionally, before the cloud obtains the interactive content information, the device further includes at least one of the following:

[0097] A first sending unit, configured to send the three-dimensional model to the first user through the cloud, and to enable the first user to edit the interactive content of the three-dimensional model;

[0098] The second sending unit is used for sending the point cloud map to the second user in the cloud, and opening the interactive content editing permission of the point cloud map for the second user.

[0099] Optionally, the first processing module 403 includes:

[0100] A first acquisition unit, configured to acquire, in the cloud, third position information of the third user in the point cloud map acquired based on the VPS technology;

[0101] A first determining unit, configured to determine, on the cloud side, relative posture information of the first interactive content with the third posture information as a reference, based on the third posture information and the first posture information;

[0102] The first processing unit is configured to display the first interactive content at a relative position of the third user's field of view according to the relative posture information.

[0103] Optionally, the second processing module 404 includes:

[0104] A second determining unit is used in the cloud to determine, based on the corresponding relationship, that the second posture information corresponds to fourth posture information in the three-dimensional model; wherein the corresponding relationship includes a one-to-one correspondence between a point position in the point cloud map and a point position in the three-dimensional model;

[0105] A second processing unit is configured to add, in the cloud, the second interactive content at a position corresponding to the fourth posture information in the three-dimensional model.

[0106] Optionally, the first interactive content or the second interactive content includes at least one of the following: image, text, audio, video, or three-dimensional model.

[0107] The interactive display device 400 based on augmented reality provided in the embodiment of the present application can realize Figure 1 , Figure 2 or Figure 3 To avoid repetition, the various processes implemented by the method embodiment will not be described again here.

[0108] like Figure 5 As shown, the embodiment of the present application also provides an electronic device, including a processor 500, a memory 520 and a transceiver 510, a program or instruction stored in the memory 520 and executable on the processor 500, and the program or instruction is executed by the processor 500 to implement the following Figure 1 , Figure 2 or Figure 3 The various processes of the embodiment of the interactive display method based on augmented reality are shown in the figure, and the same technical effect can be achieved. To avoid repetition, they will not be described here.

[0109] The embodiment of the present application also provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the above Figure 1 , Figure 2 or Figure 3 The various processes of the embodiment of the interactive display method based on augmented reality can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0110] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0111] The present application embodiment further provides a computer program / program product, wherein the computer program / program product is stored in a storage medium, and the computer program / program product is executed by at least one processor to implement the above Figure 1 , Figure 2 or Figure 3 The various processes of the embodiment of the interactive display method based on augmented reality can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0112] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for a terminal (which can be a mobile phone, computer, computing network, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0114] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An interactive display method based on augmented reality, characterized in that: include: The cloud constructs a three-dimensional model and a point cloud map of the real scene based on image data uploaded by a user in the real scene, and determines the correspondence between the three-dimensional model and the point cloud map; wherein the correspondence between the three-dimensional model and the point cloud map is a correspondence in physical space position, and the correspondence represents that the position and posture of an object in the three-dimensional model coordinate system corresponds to the position and posture of the object in the point cloud map coordinate system, and when the corresponding points in the three-dimensional model and the corresponding points in the point cloud map are in different coordinate systems, the corresponding points in the three-dimensional model and the corresponding points in the point cloud map point to the same position in the actual physical space; The cloud obtains interactive content information, and the interactive content information includes at least one of the following: first interactive content and first posture information added by a first user who is not in the real scene to the three-dimensional model, and second interactive content and second posture information added by a second user who is in the real scene to the point cloud map; wherein the first posture information includes the position coordinates and orientation angle of the first interactive content in the three-dimensional model, and the second posture information includes the position coordinates and orientation angle of the second interactive content in the point cloud map; In the case where the interactive content information includes the first interactive content and the first posture information, the cloud sends the first interactive content and the first posture information to a third user located in the real scene based on the corresponding relationship, determines the relative posture information of the third user and the first interactive content based on the first posture information, and adds the first interactive content to a corresponding position in the field of view of the third user based on the relative posture information; In the case where the interactive content information includes the second interactive content and the second posture information, the cloud adds the second interactive content at the corresponding position of the three-dimensional model based on the corresponding relationship, the second interactive content and the second posture information, and sends the three-dimensional model with the second interactive content added to a fourth user who is not in the real scene.

2. The method according to claim 1, characterized in that The correspondence between the three-dimensional model and the point cloud map includes a correspondence established in the following manner: Determining points at multiple spatial locations in the point cloud map; Determining points at a plurality of spatial locations in the three-dimensional model; Match the points at multiple spatial positions in the point cloud map with the points at multiple spatial positions in the three-dimensional model, and generate a correspondence between the three-dimensional model and the point cloud map based on the matching results, wherein the matching results represent the points in the point cloud map and the three-dimensional model that correspond to the same spatial position.

3. The method according to claim 1 or 2, characterized in that: Before the cloud obtains the interactive content information, the method further includes at least one of the following: The cloud sends the three-dimensional model to the first user, and enables the first user to edit the interactive content of the three-dimensional model; The cloud sends the point cloud map to the second user, and enables the second user to edit the interactive content of the point cloud map.

4. The method according to claim 1 or 2, characterized in that: The determining the relative posture relationship between the third user and the first interactive content based on the first posture information, and adding the first interactive content at a corresponding position in the field of view of the third user based on the relative posture relationship, includes: The cloud obtains third position information of the third user in the point cloud map based on the VPS technology; The cloud determines, based on the third posture information and the first posture information, relative posture information of the first interactive content with the third posture information as a reference; According to the relative posture information, the first interactive content is displayed at a relative position of the third user's field of view.

5. The method according to claim 1 or 2, characterized in that: The cloud adds the second interactive content at a corresponding position of the three-dimensional model based on the corresponding relationship, the second interactive content and the second posture information, including: The cloud determines, based on the corresponding relationship, that the second posture information corresponds to the fourth posture information in the three-dimensional model; wherein the corresponding relationship includes a one-to-one correspondence between the point positions in the point cloud map and the point positions in the three-dimensional model; The cloud adds the second interactive content at a position corresponding to the fourth posture information in the three-dimensional model.

6. The method according to claim 1 or 2, characterized in that: The first interactive content or the second interactive content includes at least one of the following: image, text, audio, video, and three-dimensional model.

7. An interactive display device based on augmented reality, characterized in that: include: A first acquisition module is used to construct a three-dimensional model and a point cloud map of the real scene in the cloud based on image data uploaded by a user in the real scene, and to determine the correspondence between the three-dimensional model and the point cloud map; wherein the correspondence between the three-dimensional model and the point cloud map is a correspondence in physical space position, and the correspondence represents that the position of an object in the three-dimensional model coordinate system corresponds to its position in the point cloud map coordinate system, and when the corresponding points in the three-dimensional model and the corresponding points in the point cloud map are in different coordinate systems, the corresponding points in the three-dimensional model and the corresponding points in the point cloud map point to the same position in the actual physical space; A second acquisition module is used to acquire interactive content information in the cloud, wherein the interactive content information includes at least one of the following: first interactive content and first posture information added to the three-dimensional model by a first user who is not in the real scene, and second interactive content and second posture information added to the point cloud map by a second user who is in the real scene; wherein the first posture information includes the position coordinates and orientation angle of the first interactive content in the three-dimensional model, and the second posture information includes the position coordinates and orientation angle of the second interactive content in the point cloud map; A first processing module, configured to, when the interactive content information includes the first interactive content and the first posture information, send the first interactive content and the first posture information to a third user located in the real scene based on the corresponding relationship, determine the relative posture information between the third user and the first interactive content based on the first posture information, and add the first interactive content to a corresponding position in the field of view of the third user based on the relative posture information; A second processing module is used for adding the second interactive content to the corresponding position of the three-dimensional model based on the corresponding relationship, the second interactive content and the second posture information in the cloud when the interactive content information includes the second interactive content and the second posture information, and sending the three-dimensional model with the second interactive content added to a fourth user who is not in the real scene.

8. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the interactive display method based on augmented reality as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the augmented reality-based interactive display method as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the augmented reality-based interactive display method as described in any one of claims 1 to 6.

Images