Augmented reality (AR) remote interaction method and system

Abstract

The present application provides an augmented reality (AR) remote interaction method and a system thereof. The AR remote interaction method is applied to an AR remote interaction system. The system includes a first terminal device, a server, and a second terminal device. The first terminal device creates an AR scene and sends a data packet including anchor point coordinates, AR material size, first terminal device coordinates, and a target detection frame to the server. The server generates a scene identification number based on the data packet, so that the second terminal device participates in the AR scene. If the server determines that the AR scene and the video data to be tested meet the preset consistency conditions, the first terminal device and the second terminal device can perform AR remote interaction, and the second camera set on the second terminal device obtains the video data to be tested. AR interaction can be performed without the terminal devices being in the same physical space, providing an AR remote interaction solution, broadening the application scope of AR interaction, meeting the needs of remote AR interaction, and improving the user's AR usage experience.

Description

Technical Field

[0001] The present application relates to the field of augmented reality (AR) technology, and in particular to an augmented reality (AR) remote interaction method and system thereof. Background Art

[0002] With the rapid development of Internet technology, augmented reality (AR) technology based on network technology has been widely used in more and more fields, such as mechanical assembly, equipment maintenance, structure display, and entertainment game development.

[0003] At present, the implementation of AR applications is usually hosted and parsed by servers, such as Figure 1 As shown, Figure 1 A schematic diagram of AR interaction in the prior art provided in an embodiment of the present application is shown in FIG. Figure 1 Mobile phone 1 in the scene creates an AR scene, thereby obtaining some target anchor points, and sends the target anchor points and the environment data in which they are located to the server. The server processes the received data to obtain the corresponding sparse point map and uses it as a cloud anchor point. Then mobile phone 2 in the same environment sends an interaction request to the server. The server attempts to match the visual feature data 3 with the sparse point map in the cloud anchor point according to the interaction request. If the match is successful, the resolved anchor point is set to the same position in the same environment, so that the user of mobile phone 2 can view the same AR object as in mobile phone 1, and synchronize the interaction based on this.

[0004] However, in the prior art, the terminals that implement AR interaction must be in the same physical space environment, such as the above Figure 1 In the embodiment, mobile phone 1 and mobile phone 2 must be in the same physical space environment to perform AR interaction. It can be seen that the scenes of AR interaction in the prior art have obvious limitations and cannot meet the needs of other application scenarios in actual working conditions, such as remote AR interaction, which limits the wide application of AR interaction systems and affects the user's AR interaction experience. Summary of the invention

[0005] The present application provides an augmented reality (AR) remote interaction method and system thereof, which are used to solve the technical problem that the scenarios for AR interaction in the prior art are limited, the remote AR interaction requirements cannot be met, and thus the wide application of AR interaction systems is restricted and the user AR interaction experience is affected.

[0006] In a first aspect, the present application provides an augmented reality (AR) remote interaction method, which is applied to an AR remote interaction system, wherein the AR remote interaction system includes a first terminal device, a server, and a second terminal device; the method includes:

[0007] The server receives a data packet generated by the first terminal device creating an AR scene, where the AR scene includes an initial AR object obtained by rendering an AR material, and the data packet includes anchor point coordinates, a size of the AR material, coordinates of the first terminal device, and a target detection box, where the target detection box is used to mark a target object in the AR scene;

[0008] The server generates a scene identification number according to the data packet, so that the second terminal device participates in the AR scene according to the scene identification number, and the scene identification number is used to uniquely identify the AR scene;

[0009] If the server determines that the AR scene and the video data to be tested meet a preset consistency condition, the server controls the first terminal device to perform AR remote interaction with the second terminal device, and the second terminal device obtains the video data to be tested through a second camera.

[0010] In a possible design, the server determines that the AR scene and the video data to be tested meet the preset consistency condition, including:

[0011] The server receives the video data to be measured sent by the second terminal device, and obtains the target detection frame through the scene identification number to determine the tracking object from the target object;

[0012] The server determines whether the object to be detected in the video data to be detected and the tracking object meet a preset matching condition;

[0013] If the determination result is yes, the server determines whether there is a reference plane in the video data to be tested, so that when it is determined that the reference plane exists, it is determined that the AR scene and the video data to be tested meet the preset consistency condition, and the preset consistency condition includes the preset matching condition and the reference plane.

[0014] In a possible design, the server determines whether the object to be detected in the video data to be detected and the tracked object meet the preset matching condition, including:

[0015] The server runs a preset object detector and obtains an initial bounding box to be measured in an initial frame of the video data to be measured, wherein the initial bounding box to be measured is used to mark the object to be measured in the initial frame;

[0016] Acquire subsequent bounding boxes to be measured in subsequent frames of the video data to be measured, so as to gradually iterate the initial bounding box to be measured through the subsequent bounding boxes to be measured;

[0017] Determine the similarity between the target detection box marking the tracked object and each bounding box to be detected, and end the step-by-step iteration when the similarity is less than a preset similarity threshold to determine that the tracked object and the object to be detected meet the preset matching condition.

[0018] In a possible design, after determining that the AR scene and the video data to be tested meet the preset consistency condition, the method further includes:

[0019] The server sends an interaction instruction to the second terminal device, so that the second terminal device renders the AR material on the reference plane according to the interaction instruction to obtain a participating AR object, where the participating AR object is used to interact with the initial AR object of the first terminal device.

[0020] In a possible design, after controlling the first terminal device to perform AR remote interaction with the second terminal device, the method further includes:

[0021] The server receives the respective newly added data packets sent by the first terminal device and / or the second terminal device in real time, and updates the cloud anchor point according to the received newly added data packets;

[0022] Among them, the first terminal device and / or the second terminal device compares their respective local anchor points with the cloud anchor point according to their respective preset time periods, and when the first terminal device and / or the second terminal device determines that there is a new anchor point, they respectively send modification requests to the server.

[0023] In a possible design, after sending the modification request to the server, the method further includes:

[0024] The server generates idle state feedback information according to the modification request, and transmits the idle state feedback information back to the first terminal device or the second terminal device, so that the first terminal device or the second terminal device that receives the idle state feedback information modifies the local anchor point.

[0025] In a possible design, the first terminal device also includes a first camera, which is used to obtain environmental video data of a space in which the first terminal device is located, and the environmental video data is used to obtain the target detection frame.

[0026] In a second aspect, the present application provides an augmented reality (AR) remote interaction system, including: a first terminal device, a server, and a second terminal device;

[0027] The first terminal device is used to create an AR scene, and send a data packet generated by creating the AR scene to a server, wherein the AR scene includes an initial AR object obtained by rendering an AR material, and the data packet includes anchor point coordinates, a size of the AR material, coordinates of the first terminal device, and a target detection box, and the target detection box is used to mark a target object in the AR scene;

[0028] The server is used to generate a scene identification number according to the data packet sent by the first terminal device, so that the second terminal device participates in the AR scene according to the scene identification number, and the scene identification number is used to uniquely identify the AR scene;

[0029] The server is further configured to control the first terminal device to perform AR remote interaction with the second terminal device if it is determined that the AR scene and the video data to be tested meet a preset consistency condition;

[0030] The second terminal device includes a second camera, and the second camera is used to obtain the video data to be tested.

[0031] In one possible design, the server includes:

[0032] A first processing module, configured to receive the video data to be measured sent by the second terminal device, and obtain the target detection frame through the scene identification number to determine the tracking object from the target object;

[0033] A second processing module is used to determine whether the object to be detected in the video data to be detected and the tracking object meet a preset matching condition;

[0034] If the determination result is yes, the second processing module is further used to determine whether there is a reference plane in the video data to be tested, so that when it is determined that the reference plane exists, it is determined that the AR scene and the video data to be tested meet the preset consistency condition, and the preset consistency condition includes the preset matching condition and the reference plane.

[0035] In a possible design, the second processing module is specifically used to:

[0036] Running a preset object detector and obtaining an initial bounding box to be measured in an initial frame of the video data to be measured, wherein the initial bounding box to be measured is used to mark the object to be measured in the initial frame;

[0037] Acquire subsequent bounding boxes to be measured in subsequent frames of the video data to be measured, so as to gradually iterate the initial bounding box to be measured through the subsequent bounding boxes to be measured;

[0038] Determine the similarity between the target detection box marking the tracked object and each bounding box to be detected, so that when the similarity is less than a preset similarity threshold, end the step-by-step iteration to determine that the tracked object and the object to be detected meet the preset matching condition.

[0039] In one possible design, the server further includes:

[0040] A first sending module, used for sending an interaction instruction to the second terminal device;

[0041] The second terminal device also includes:

[0042] The third processing module is used to render the AR material on the reference plane according to the interaction instruction to obtain a participating AR object, where the participating AR object is used to interact with the initial AR object of the first terminal device.

[0043] In a possible design, the first terminal device and / or the second terminal device further includes:

[0044] A second sending module, configured to send respective newly added data packets to the server in real time, so that the server updates the cloud anchor point according to the received newly added data packets;

[0045] a fourth processing module, configured to compare respective local anchor points with the cloud anchor points according to respective preset time periods;

[0046] If it is determined that there is a newly added anchor point, the second sending module is further configured to send a modification request to the server respectively.

[0047] In one possible design, the server further includes:

[0048] The fifth processing module is used to generate idle state feedback information according to the modification request, and transmit the idle state feedback information back to the first terminal device or the second terminal device, so that the first terminal device or the second terminal device that receives the idle state feedback information modifies the local anchor point.

[0049] In a possible design, the first terminal device further includes:

[0050] The first camera is used to obtain environmental video data of the space where the first terminal device is located, and the environmental video data is used to obtain the target detection frame.

[0051] In a third aspect, the present application provides an augmented reality (AR) remote interaction system, comprising:

[0052] Memory for storing computer programs;

[0053] The processor is used to call and execute the computer program in the memory to perform any possible AR remote interaction method provided in the first aspect.

[0054] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to execute any possible AR remote interaction method provided in the first aspect.

[0055] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any possible AR remote interaction method provided in the first aspect.

[0056] The present application provides an augmented reality AR remote interaction method and system thereof. The AR remote interaction method provided in the present application is applied to an AR remote interaction system, wherein the AR remote interaction system includes a first terminal device, a server and a second terminal device. First, the first terminal device creates an AR scene and sends a data packet including anchor point coordinates, AR material size, first terminal device coordinates and a target detection frame to the server, wherein the AR scene includes an initial AR object obtained after rendering the AR material, and the target detection frame is used to mark the target object in the AR scene. The server generates a scene identification number according to the received data packet, so that the second terminal device participates in the AR scene through the scene identification number. Afterwards, if the server determines that the AR scene and the video data to be tested meet the preset consistency condition, it is determined that the first terminal device and the second terminal device can perform AR remote interaction, wherein the video data to be tested is obtained by the second terminal device through the second camera set thereon. When the AR scene and the video data to be tested meet the preset consistency condition, the AR remote interaction between the first terminal device and the second terminal device can be realized, without the need for the terminal devices to be in the same physical space for AR interaction, so that the implementation scheme of AR interaction is effectively expanded, further meeting the remote AR interaction requirements, and improving the user AR usage experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0058] Figure 1 A schematic diagram of AR interaction in the prior art provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0060] Figure 3 A flowchart of an AR remote interaction method provided in an embodiment of the present application;

[0061] Figure 4 A flowchart of another AR remote interaction method provided in an embodiment of the present application;

[0062] Figure 5 A flowchart of another AR remote interaction method provided in an embodiment of the present application;

[0063] Figure 6 A flowchart of another AR remote interaction method provided in an embodiment of the present application;

[0064] Figure 7 A schematic diagram of the structure of an AR remote interaction system provided in an embodiment of the present application;

[0065] Figure 8 A schematic diagram of the structure of another AR remote interaction system provided in an embodiment of the present application;

[0066] Fig. 9 A schematic diagram of the structure of another AR remote interaction system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of methods and devices consistent with some aspects of the present application as detailed in the appended claims.

[0068] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0069] In the prior art, terminal devices for AR interaction must be in the same physical space environment, which makes the AR interaction scene have obvious limitations and cannot meet the AR interaction needs of other application scenarios in actual working conditions such as remote control, limiting the widespread application of AR interaction systems and affecting users' AR interaction experience.

[0070] In response to the above-mentioned problems in the prior art, the present application provides an augmented reality AR remote interaction method and system thereof. The AR remote interaction method provided in the present application can be applied to the AR remote interaction system provided in the present application. The inventive concept of the present application is that the data packet sent by the first terminal device to the server contains a target detection frame that can mark the target object in the AR scene, and the server determines the tracking object from the target object. When the second terminal device participates in the AR scene, the server determines whether the AR scene and the video data obtained by the second terminal device, that is, the video data to be tested, meet the preset consistency conditions. If so, the first terminal device and the second terminal device can perform AR remote interaction. The implementation of AR remote interaction between terminal devices is based on whether the AR scene and the video data to be tested meet the preset consistency conditions, and there is no restriction on the spatial scene in which the terminal device is located, thereby providing a solution for AR remote interaction between terminal devices.

[0071] The following is an introduction to exemplary application scenarios of the embodiments of the present application.

[0072] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present application. Figure 2 As shown, the augmented reality (AR) interaction method provided in the embodiment of the present application can be applied to an AR remote interaction system, which includes a terminal device 21, a server 22, and a terminal device 23. The network is used to provide a medium for the communication link between the terminal device 21, the server 22, and the terminal device 23. The network may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc. The terminal device 21, the server 22, and the terminal device 23 can interact through the network to receive or send messages, thereby realizing the AR remote interaction method provided in the embodiment of the present application. Among them, one of the terminal device 21 and the terminal device 23 can be the first terminal device in the embodiment of the present application, and the other can be the second terminal device accordingly. For example, when the terminal device 21 is the first terminal device, the terminal device 23 is the second terminal device. It is worth noting that the number of the second terminal devices can be one or more, Figure 2The number of terminal devices 23 shown as the second terminal device is one. In addition, the terminal devices 21 and 23 can be any terminal devices that can run AR corresponding applications, such as smart phones, smart glasses, smart bracelets, smart watches, tablet computers, etc. The type of terminal devices is not limited in this embodiment of the application. Figure 2 The terminal device 21 and the terminal device 23 are both shown as smartphones. The server 22 may also be a server cluster, which is not limited in this embodiment. In addition, the terminal devices for performing AR remote interaction provided in the embodiment of the present application may be in the same or different physical space scenes, which is not limited.

[0073] It should be noted that the above application scenarios are merely illustrative, and the augmented reality (AR) remote interaction method and system provided in the embodiments of the present application include but are not limited to the above application scenarios.

[0074] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0075] Figure 3 A flowchart of an AR remote interaction method provided in an embodiment of the present application is provided. The AR remote interaction method provided in this embodiment can be applied to an AR interaction system, wherein the AR interaction system includes a first terminal device, a server, and a second terminal device, wherein the number of the first terminal device is one, and the number of the second terminal devices can be multiple. Figure 3 The number of second terminal devices is used as a schematic illustration. Figure 3 As shown, the AR remote interaction method provided in this embodiment includes:

[0076] S101: The first terminal device creates an AR scene, and sends a data packet generated by creating the AR scene to a server.

[0077] Correspondingly, the server receives the data packet generated by the first terminal device to create the AR scene.

[0078] Among them, the AR scene includes an initial AR object obtained after rendering the AR material, and the data packet includes anchor point coordinates, the size of the AR material, the coordinates of the first terminal device and the target detection frame, and the target detection frame is used to mark the target object in the AR scene.

[0079] The first terminal device creates an AR scene, that is, the first terminal device is the initiator of the AR remote interaction.

[0080] By clicking the anchor point on the first terminal device, a point in the actual physical space can be defined, and the coordinates of this point in the actual physical space are the anchor point coordinates. The size of the AR material can be any size set for the AR material after the first terminal device runs the preset AR program to select the AR material. In other words, the first terminal device can enlarge or reduce the size of the selected AR material by itself. The coordinates of the first terminal device are the actual coordinates of the first terminal device in the actual physical space, such as the longitude and latitude of the first terminal device in the actual physical space. The target detection box is used to mark the target object in the AR scene. The target object can be any object other than the non-initial AR object in the AR scene, such as animals, people, objects, etc.

[0081] The process of the first terminal device creating an AR scene is described as follows.

[0082] For example, the first terminal device first runs a preset AR program configured in itself, such as running an ARCore application software, and then obtains a plane within the field of view in the actual physical space where the first terminal device is located through the plane detection function in the preset AR program. The obtained plane can be any plane such as a desktop or the ground, so that the initial AR object will not appear in the air, and then the selected AR material is placed on the grid area rendered by running the preset AR program, and then the relative position of the two is determined by using the anchor point coordinates and the coordinates of the first terminal device, and then the AR material is rendered according to the selected AR material and the set size of the AR material, so as to obtain the initial AR object. Further, the first terminal device is configured with a first camera, and the first camera uses the initial AR object as a reference object to obtain the environmental video data of the space where the first terminal device is located around the initial AR object, and the space is the actual physical space. For example, with the initial AR object as the center, the first camera shoots around the initial AR object in the space where the first terminal device is located, and the captured video is the environmental video data of the space where the first terminal device is located. The creation of the AR scene can be completed by placing the initial AR object, a virtual object, in the actual physical space, and this can be achieved by running a preset AR program.

[0083] In addition, the target detection frame is used to mark the target object in the AR scene. For example, the target object in the AR scene is identified by a preset target detection algorithm and marked with a labeling frame, thereby obtaining the target detection frame.

[0084] The target detection frame can be obtained by performing target recognition on the environmental video data obtained by the first terminal device through the first camera. Specifically, the candidate area containing any object in the environmental video data can be determined first by a preset search algorithm (Selective Search), and then the picture where the candidate area is located is input into a preset convolutional neural network, such as CNN (Convolutional Neural Network), to identify each object in the input picture, and mark the identified object through the candidate frame. Further, feature extraction is performed on the candidate frame, and each feature is identified by a preset classifier, and the candidate frames whose features belong to the same category are classified into one category. Furthermore, the identified candidate frames of the same category can also be optimized in terms of parameters such as position, and the optimized candidate frame is determined as the target detection frame. Among them, the object marked by the target detection frame is the target object. The preset classifier in this embodiment can be any corresponding algorithm that can achieve classification, and this embodiment is not limited to this.

[0085] After the first terminal device creates the AR scene, the data such as the anchor point coordinates, the size of the AR material, the first terminal device coordinates, and the target detection frame described in the above embodiment are sent to the server in the form of a data packet.

[0086] S102: The server generates a scene identification number according to the data packet, so that the second terminal device participates in the AR scene according to the scene identification number.

[0087] The scene identification number is used to uniquely identify the AR scene.

[0088] After receiving the data packet, the server generates a scene identification number according to the data packet to uniquely identify the AR scene through the scene identification number, that is, different scene identification numbers are generated for different first terminal devices, and different scene identification numbers are generated for data packets sent by the same first terminal device at different times, regardless of whether the data packets are the same. For example, the scene identification number can be formed by the MAC address of the first terminal device and a random number to have unique identification.

[0089] After the server generates the scene identification number, the second terminal device can also participate in the AR scene according to the scene identification number by running the preset AR program. For example, the second terminal device accesses the server and initiates a query request to the server to query whether the scene identification number already exists in the server. If it does, it means that an AR scene has been created by a terminal device, and the second terminal device can participate in the corresponding AR scene by selecting the scene identification number. When the second terminal device accesses the server and the scene identification number has not been generated in the server, the second terminal device can choose to exit the preset AR program or change its role to create an AR scene. Among them, the scene identification number generated by the server can be displayed in the form of a list.

[0090] S103: The server determines whether the AR scene and the video data to be tested meet a preset consistency condition.

[0091] The second terminal device obtains the video data to be tested through the second camera.

[0092] After the second terminal device participates in the AR scene, the server determines whether the AR scene and the video data to be tested meet the preset consistency condition. If so, it is determined that the AR scene and the video data to be tested meet the preset consistency condition, and the first terminal device and the second terminal device can realize AR remote interaction, and step S104 is executed. On the contrary, if not, it is determined that the AR scene and the video data to be tested do not meet the preset consistency condition, and the first terminal device and the second terminal device cannot realize AR remote interaction.

[0093] The video data to be tested is acquired by collecting the second camera set on the second terminal device after the second terminal device participates in the AR scene. The second camera can be any camera set on the second terminal device. There is no limitation on the way the second camera acquires the video data to be tested.

[0094] In addition, in this step, the implementation method in which the server determines whether the AR scene and the video data to be tested meet the preset consistency condition can also be configured on a first terminal device or a second terminal device with excellent performance, and this embodiment does not limit this.

[0095] S104: If the determination result is yes, control the first terminal device to perform AR remote interaction with the second terminal device.

[0096] On the contrary, that is, the determination result is no, the first terminal device and the second terminal device cannot perform AR remote interaction.

[0097] A terminal device is in the same actual physical space as other terminal devices. In this actual physical space, there must be the same target object for these terminal devices, so AR interaction can be performed between the terminal devices. When at least two terminal devices are in completely different physical spaces, it is necessary to determine whether the AR scene and the video data to be tested meet the preset consistency conditions. If they meet the conditions, the server can control the AR remote interaction between the terminal devices. Otherwise, AR remote interaction cannot be performed.

[0098] The AR remote interaction method provided in the embodiment of the present application is applied to an AR remote interaction system, wherein the AR remote interaction system includes a first terminal device, a server, and a second terminal device. First, the first terminal device creates an AR scene and sends a data packet generated by creating the AR scene to the server, wherein the data packet includes the anchor point coordinates, the size of the AR material, the coordinates of the first terminal device, and the target detection frame, and the target detection frame is used to mark the target object in the AR scene. Then, the server generates a scene identification number according to the received data packet, so that the second terminal device participates in the AR scene through the scene identification number. Furthermore, when the server determines that the AR scene and the video data to be tested meet the preset consistency condition, the first terminal device and the second terminal device are controlled to realize AR remote interaction. Among them, the video data to be tested is obtained by the second terminal device through the second camera. Thus, AR interaction can be performed without the terminal devices being in the actual physical space of the same geographical location, providing an AR remote interaction solution, effectively broadening the application scope of AR interaction, meeting the needs of remote AR interaction, and improving the user AR usage experience.

[0099] In a possible design, a possible implementation of step S103 in the above embodiment is as follows: Figure 4 As shown, Figure 4 A flowchart of another AR remote interaction method provided in an embodiment of the present application. Figure 4 As shown, in the AR remote interaction method provided in this embodiment, the server determines whether the AR scene and the video data to be tested meet the preset consistency condition, including:

[0100] S201: The server receives the video data to be tested sent by the second terminal device, and obtains the target detection frame through the scene identification number to determine the tracking object from the target object marked in the target detection frame.

[0101] When the server is configured to determine whether the AR scene and the video data to be tested meet the preset consistency conditions, the server receives the video data to be tested sent by the second terminal device, and at the same time, obtains the target detection frame in the corresponding data packet through the scene identification number selected by the second terminal device when participating in the AR scene. After that, the server determines a tracking object from the target objects marked by the acquired target detection frame. Specifically, one or more target objects can be selected as tracking objects from all target objects marked by the target detection frame according to the actual working conditions. Thus, the server completes the reception of the video data to be tested sent by the second terminal device, and obtains the target detection frame through the scene identification number to determine the tracking object.

[0102] S202: The server determines whether the object to be detected in the video data to be detected and the tracking object meet a preset matching condition.

[0103] After receiving the video data to be tested and determining the tracking object, the server further determines whether the object to be tested in the video data to be tested and the tracking object meet the preset matching conditions. For example, a target tracking module may be preset in the server to determine whether the object to be tested and the tracking object meet the preset matching conditions through the preset target tracking module. The object to be tested is any object in the video data to be tested, such as an animal, a person, an object, etc. Obviously, this step is essentially to determine whether the object to be tested in the video data to be tested is the same as the tracking object determined in the previous step. If they are the same, the result is determined to be yes, the preset matching conditions are met, and S203 is further executed. Otherwise, the preset matching conditions are not met, and S204 is executed.

[0104] It should be noted that the target tracking module preset in the server can be any module that can realize target tracking and identification, and this embodiment does not limit this.

[0105] S203: The server determines whether there is a reference plane in the video data to be tested, so as to determine whether the AR scene and the video data to be tested meet a preset consistency condition when it is determined that there is a reference plane.

[0106] The preset consistency condition includes a preset matching condition and a reference plane.

[0107] When the server determines that the object to be tested and the tracked object meet the preset matching conditions, the server further determines whether there is a reference plane in the video data to be tested. If so, when it is determined that there is a reference plane, it is determined that the AR scene and the video data to be tested meet the preset consistency conditions. On the contrary, if it does not exist, the server ends the determination of whether the AR scene and the video data to be tested meet the preset consistency conditions. It can be understood that the preset consistency conditions include the preset matching conditions and the presence of a reference plane in the video data to be tested. Among them, there is no limitation on the reference plane, and it can be any horizontal plane.

[0108] S204: The server ends determining whether the AR scene and the video data to be tested meet a preset consistency condition.

[0109] If it is determined that the object to be measured and the tracked object in the video data to be measured do not meet the preset matching condition, the current determination process is terminated, that is, the step of determining whether the AR scene and the video data to be measured meet the preset consistency condition is terminated.

[0110] In the AR remote interaction method provided by the embodiment of the present application, when the server determines whether the AR scene and the video data to be tested meet the preset consistency condition, the server first receives the video data to be tested sent by the second terminal device, obtains the target detection frame through the scene identification number, and determines the tracking object from the target object. Then the server determines whether the object to be tested and the tracking object in the video data to be tested meet the preset matching condition. If the determination result is yes, the server further determines whether there is a reference plane in the video data to be tested, and when it is determined that there is, it is determined that the AR scene and the video data to be tested meet the preset consistency condition. On the other hand, if it is determined that the object to be tested and the tracking object do not meet the preset matching condition, or there is no reference plane in the video data to be tested, the server ends the step of determining whether the AR scene and the video data to be tested meet the preset consistency condition. The server determines whether the AR scene and the video data to be tested meet the preset consistency condition by determining whether the object to be tested and the tracking object meet the preset matching condition and whether there is a reference plane. When the two meet the preset consistency condition, the first terminal device and the second terminal device realize AR remote interaction, which effectively broadens the application scope of AR interaction, further meets the remote AR interaction needs, and improves the user AR usage experience.

[0111] In a possible design, the possible implementation of step S102 is as follows: Figure 5 As shown, Figure 5 A flowchart of another AR remote interaction method provided in an embodiment of the present application. Figure 5 As shown, in the AR remote interaction method provided in this embodiment, the server determines whether the object to be measured in the video data to be measured and the tracking object meet the preset matching condition, including:

[0112] S1021: The server runs a preset object detector and obtains an initial bounding box to be tested in an initial frame of the video data to be tested.

[0113] The initial bounding box to be measured is used to mark the object to be measured in the initial frame.

[0114] The server runs a preset object detector, for example, a preset target tracking module can be used to obtain an initial bounding box to be measured in the initial frame corresponding to the video data to be measured, so as to mark the object to be measured in the initial frame with the initial bounding box to be measured. Among them, both the preset object detector and the preset target tracking module can be realized by running the corresponding target detection software configured in the server, and the embodiment of the present application does not limit the specific application of the target detection software. The initial frame can be understood as any frame selected from the video data to be measured, and the video data to be measured is processed starting from this frame, and therefore, it is defined as the initial frame.

[0115] S1022: Obtain subsequent bounding boxes to be tested in subsequent frames of the video data to be tested, so as to gradually iterate the initial bounding box to be tested through the subsequent bounding boxes to be tested.

[0116] After obtaining the initial bounding box to be measured corresponding to the object to be measured in the initial frame, further obtain the subsequent bounding boxes to be measured for the object to be measured in the subsequent frames after the initial frame in the video data to be measured, and gradually iterate the initial bounding box to be measured through the subsequent bounding boxes to be measured. For example, the preset target tracking module can be offset in the direction of the maximum probability density of the object to be measured, and the object to be measured captured in the subsequent frames is continuously marked with the bounding box to be measured, so as to achieve the purpose of gradually iterating the initial bounding box to be measured through the subsequent bounding boxes to be measured. It is worth noting that the embodiment of the present application does not limit the specific step-by-step iteration method, and the corresponding software program can be selected according to the actual working conditions to achieve the purpose of step-by-step iteration.

[0117] S1023: Determine the similarity between the target detection box annotating the tracked object and each bounding box to be measured, and when the similarity is less than a preset similarity threshold, end the step-by-step iteration to determine that the tracked object and the object to be measured meet a preset matching condition.

[0118] In the above embodiment, the initial bounding box to be measured is gradually iterated through subsequent bounding boxes to be measured. In the step-by-step iterative process, each time the bounding box to be measured is obtained, that is, each bounding box to be measured obtained, the similarity between the bounding box to be measured and the target detection box is determined. When the determined similarity is less than a preset similarity threshold, the step-by-step iterative process is terminated, indicating that the object to be measured that is consistent with the target object has been tracked, and it is determined that the tracked object and the object to be measured meet the preset matching condition.

[0119] On the contrary, if the similarity between the target detection frame and the bounding frame to be tested is not less than the preset similarity threshold, the subsequent bounding frames to be tested will continue to be obtained, that is, the step-by-step iteration process will continue until the similarity is less than the preset similarity threshold, and the step-by-step iteration will end. However, there is a possibility that when the step-by-step iteration reaches the last frame of the video data, the similarity between the obtained bounding frame to be tested and the target detection frame of the marked tracked object is still not less than the preset similarity threshold, indicating that there is no object to be tested that is the same as the target object in the video data to be tested, and then it is determined that the object to be tested and the tracked object do not meet the preset matching condition.

[0120] In the AR remote interaction method provided by the embodiment of the present application, when the server determines whether the object to be tested and the tracked object in the video data to be tested meet the preset matching conditions, the server first runs the preset object detector and obtains the initial bounding box to be tested used to mark the object to be tested in the initial frame corresponding to the video data to be tested, and then obtains the subsequent bounding box to be tested in the subsequent frame, and gradually iterates the initial bounding box to be tested through the subsequent bounding boxes to be tested, until the similarity between the target detection box of the marked target object and the bounding box to be tested is less than the preset similarity threshold, the step-by-step iteration process is ended, and it is determined that the tracked object and the object to be tested meet the preset matching conditions. When the similarity between the bounding box to be tested obtained in the last frame of the video data and the target detection box of the marked target object is still not less than the preset similarity threshold, it indicates that there is no object to be tested that is the same as the target object in the video data to be tested, and the object to be tested and the tracked object do not meet the preset matching conditions. Thus, through a step-by-step iteration method, it can be determined whether the object to be measured and the tracked object meet the preset matching conditions, and when the conditions are met and the reference plane is obtained, AR remote interaction is realized between the first terminal device and the second terminal device, enriching the implementation scheme of AR interaction, meeting the needs of remote AR interaction, and improving the user's AR usage experience.

[0121] In a possible design, after determining that the AR scene and the video data to be tested meet the preset consistency condition, the server sends an interaction instruction to the second terminal device, the second terminal device receives the interaction instruction, and renders the AR material on the reference plane according to the interaction instruction to obtain the participating AR object, so that the second terminal device interacts with the initial AR object of the first terminal device through the participating AR object, thereby realizing the AR remote interaction between the control terminal devices. Among them, while receiving the interaction instruction, the second terminal device can also obtain the data packet sent by the first terminal device to the server according to the interaction instruction, and obtain the size of the AR material in the data packet, so as to render the AR material.

[0122] After the first terminal device and the second terminal device perform AR remote interaction, since each terminal device can continue to obtain corresponding video data according to its own camera, in a possible design, Figure 6 A flowchart of another AR remote interaction method provided in an embodiment of the present application. Figure 6 As shown, in the AR remote interaction method provided in this embodiment, after controlling the first terminal device to perform AR remote interaction with the second terminal device, it also includes:

[0123] S301: The server receives new data packets respectively sent by the first terminal device and / or the second terminal device in real time, and updates the cloud anchor point according to the received new data packets.

[0124] One or both of them will send their own newly added data packets to the server in real time. The newly added data packets may include the newly acquired anchor point coordinates, the size of the AR material, their own coordinates and other data. After receiving the newly added data packet sent by the terminal device, the server will update the cloud anchor point according to the newly added data packet. The cloud anchor point can be understood as the anchor point stored in the server. The data included in the newly added data packet is determined by the time node when the first terminal device and the second terminal device realize AR remote interaction.

[0125] S302: The first terminal device and / or the second terminal device compares their respective local anchor points with the cloud anchor points according to their respective preset time periods.

[0126] The first terminal device and / or the second terminal device will compare their local anchor points with the cloud anchor points in the server according to their respective preset time periods to determine whether to modify and update their local anchor points. The durations corresponding to the preset time periods of the first terminal device and the second terminal device can be the same or different, and this is not limited in the embodiments of the present application.

[0127] S303: When the first terminal device and / or the second terminal device determines that there is a newly added anchor point, they respectively send a modification request to the server.

[0128] The first terminal device and / or the second terminal device determines that there is a new anchor point in the cloud anchor point by comparing the local anchor point with the cloud anchor point, and then sends a modification request to the server respectively to modify their respective local anchor points to keep them synchronized with the cloud anchor point.

[0129] Accordingly, after the first terminal device and / or the second terminal device compare, if it is determined that there is no newly added anchor point, the current state of each is maintained without sending a modification request to the server. After the first terminal device and / or the second terminal device sends a modification request to the server, the server determines which terminal device is currently allowed to modify its own local anchor point. In a possible design, the server first generates idle state feedback information according to the modification request, and transmits the idle state feedback information back to the first terminal device or the second terminal device. Accordingly, the terminal device that receives the idle state feedback information, that is, the first terminal device or the second terminal device that receives the idle state feedback information, modifies the local anchor point so that the local anchor point is synchronized with the cloud anchor point. In other words, the server only allows one terminal device to modify the local anchor point at present. When a terminal device is modifying its own local anchor point, the other terminal device can modify its own local anchor point only after the terminal device that is modifying the local anchor point has finished modifying it. It is understandable that the first terminal device or the second terminal device can learn whether there is a terminal device currently modifying the local anchor point by querying the server.

[0130] In the AR remote interaction method provided by the embodiment of the present application, after the first terminal device and the second terminal perform AR remote interaction, the first terminal device and / or the second terminal device will send their respective newly added data packets to the server in real time, so that the server updates the cloud anchor point according to the received data packets. At the same time, the first terminal device and / or the second terminal device will compare their respective local anchor points with the cloud anchor points according to their respective preset time periods to determine whether there is a newly added anchor point. If it is determined that there is, a modification request will be sent to the server respectively, so that the server allows itself to modify the local anchor point, so that the modified local anchor point is synchronized with the cloud anchor point. Thereby, it can ensure that the AR remote interaction between the first terminal device and the second terminal device can be carried out smoothly, and improve the user's AR usage experience.

[0131] The following are device embodiments of the present application, which can be used to execute the corresponding method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the corresponding method embodiments of the present application.

[0132] Figure 7 This is a schematic diagram of the structure of an AR remote interaction system provided in an embodiment of the present application. Figure 7 As shown, the AR remote interaction system 400 provided in this embodiment includes: a first terminal device 401, a server 402 and a second terminal device 403.

[0133] The first terminal device 401 is used to create an AR scene and send a data packet generated by creating the AR scene to a server.

[0134] The AR scene includes an initial AR object obtained by rendering the AR material, and the data packet includes the anchor point coordinates, the size of the AR material, the first terminal device coordinates, and the target detection frame, where the target detection frame is used to mark the target object in the AR scene;

[0135] The first terminal device 401 further includes:

[0136] The first camera 4011 is used to obtain environmental video data of the space where the first terminal device 401 is located, so as to obtain a target detection frame through the environmental video data;

[0137] The server 402 is used to generate a scene identification number according to the data packet, and control the second terminal device to participate in the AR scene according to the scene identification number;

[0138] Among them, the scene identification number is used to uniquely identify the AR scene;

[0139] The server 402 is further configured to enable the first terminal device to perform AR remote interaction with the second terminal device if it is determined that the AR scene and the video data to be tested meet a preset consistency condition;

[0140] The second terminal device 403 includes:

[0141] The second camera 4031 is used to obtain the video data to be tested.

[0142] exist Figure 7 Based on the embodiment shown, Figure 8 This is a schematic diagram of the structure of another AR remote interaction system provided in an embodiment of the present application. Figure 8 As shown, the server 402 in the AR remote interaction system 400 provided in this embodiment includes:

[0143] The first processing module 4021 is used to receive the video data to be tested sent by the second terminal device 403, and obtain the target detection frame through the scene identification number to determine the tracking object from the target object;

[0144] The second processing module 4022 is used to determine whether the object to be detected and the tracking object in the video data to be detected meet a preset matching condition;

[0145] If the determination result is yes, the second processing module 4022 is further used to determine whether there is a reference plane in the video data to be tested, so that when it is determined that there is a reference plane, it is determined that the AR scene and the video data to be tested meet a preset consistency condition, and the preset consistency condition includes a preset matching condition and a reference plane.

[0146] In a possible design, the second processing module 4022 is specifically configured to:

[0147] Running a preset object detector and obtaining an initial bounding box to be tested in an initial frame of the video data to be tested, where the initial bounding box to be tested is used to mark the object to be tested in the initial frame;

[0148] Acquire subsequent bounding boxes to be tested in subsequent frames of the video data to be tested, so as to gradually iterate the initial bounding box to be tested through the subsequent bounding boxes to be tested;

[0149] Determine the similarity between the target detection box annotating the tracked object and each bounding box to be measured, and when the similarity is less than a preset similarity threshold, end the step-by-step iteration to determine that the tracked object and the object to be measured meet the preset matching condition.

[0150] Continue to refer to Figure 8 In one possible design, the server 402 further includes:

[0151] A first sending module 4023, configured to send an interaction instruction to the second terminal device 403;

[0152] The second terminal device 403 further includes:

[0153] The third processing module 4032 is used to render the AR material on the reference plane according to the interaction instruction to obtain the participating AR object, which is used to interact with the initial AR object of the first terminal device 401.

[0154] Continue to refer to Figure 8 As shown, in a possible design, the first terminal device 401 and / or the second terminal device 403 further includes:

[0155] The second sending module 404 is used to send the respective newly added data packets to the server 402 in real time, so that the server 402 updates the cloud anchor point according to the received newly added data packets;

[0156] The fourth processing module 405 is used to compare the respective local anchor points with the cloud anchor points according to respective preset time periods;

[0157] If it is determined that there is a newly added anchor point, the second sending module 404 is further configured to send a modification request to the server 402 .

[0158] It should be noted that Figure 8 In the example, the first terminal device 401 and the second terminal device 403 both include the second sending module 404 and the fourth processing module 405. Optionally, the first terminal device 401 or the second terminal device 403 may include the second sending module 404 and the fourth processing module 405, which is not limited in this embodiment.

[0159] Continue to refer to Figure 8 As shown, optionally, the server 402 further includes:

[0160] The fifth processing module 4024 is used to generate idle state feedback information according to the modification request, and transmit the idle state feedback information back to the first terminal device 401 or the second terminal device 403, so that the first terminal device 401 or the second terminal device 403 that receives the idle state feedback information modifies the local anchor point.

[0161] The above-mentioned AR remote interaction system embodiment provided in this application is only schematic, and the module division therein is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another system. The coupling between the modules can be achieved through some interfaces, which are usually electrical communication interfaces, but it is not ruled out that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices.

[0162] It is worth noting that the first terminal device, server and second terminal device in the AR remote interaction system provided in the above-mentioned embodiment can be used to execute the corresponding steps of the AR remote interaction method provided in the above-mentioned embodiment. The specific implementation method, principle and technical effect are similar to those of the aforementioned method embodiment and will not be repeated here.

[0163] Fig. 9 This is a schematic diagram of the structure of another AR remote interaction system provided in an embodiment of the present application. Fig. 9 As shown, the AR remote interaction system 500 provided in this embodiment includes:

[0164] A first camera 501 and a second camera 502;

[0165] Processor 503 and memory 504 .

[0166] The memory 504 is used to store the program of the processor 503. Specifically, the program may include program code, and the program code includes computer operation instructions.

[0167] The memory 504 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0168] The processor 503 is configured to execute the computer program stored in the memory 504 to implement the corresponding steps in the AR remote interaction method in the above method embodiments. The processor 503 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0169] In addition, it should be noted that the processor 503 listed in this embodiment is a general term for the processors configured in each terminal device and server included in the AR remote interaction system 500, and the processors configured in each terminal device and server are used to execute their respective computer programs to implement the corresponding steps in the AR remote interaction method provided in the method embodiment of the present application.

[0170] Optionally, the memory 504 may be independent or integrated with the processor 503. When the memory 504 is a device independent of the processor 503, the AR remote interaction system 500 may further include:

[0171] The bus 505 is used to connect the processor 503 and the memory 504. The bus may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0172] Optionally, in a specific implementation, if the processor 503 and the memory 504 are integrated on a chip, the processor 503 and the memory 504 can communicate through an internal interface.

[0173] Understandably, Fig. 9 The illustrated embodiment is a corresponding electronic device in an AR remote interaction system.

[0174] The present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes. Specifically, a computer program is stored in the computer-readable storage medium. When the processor executes the computer program, the corresponding electronic device executes the corresponding steps of the AR remote interaction method provided in the above-mentioned various embodiments.

[0175] The embodiment of the present application also provides a computer program product, which includes a computer program stored in a readable storage medium. The processor can read the computer program from the readable storage medium and execute the computer program so that the corresponding electronic device implements the corresponding steps of the AR remote interaction method provided in the above various embodiments.

[0176] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0177] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An augmented reality (AR) remote interaction method, characterized in that: Applied to an AR remote interaction system, the AR remote interaction system includes a first terminal device, a server and a second terminal device; the method includes: The server receives a data packet generated by the first terminal device when creating an AR scene, wherein the AR scene includes an initial AR object obtained by rendering an AR material, and the data packet includes anchor point coordinates, a size of the AR material, coordinates of the first terminal device, and a target detection frame, where the target detection frame is used to mark the target object in the AR scene; the first terminal device obtains environmental video data of a space where the first terminal device is located through a first camera, and the environmental video data is used to obtain the target detection frame; The server generates a scene identification number according to the data packet, so that the second terminal device participates in the AR scene according to the scene identification number, and the scene identification number is used to uniquely identify the AR scene; If the server determines that the AR scene and the video data to be tested meet a preset consistency condition, the server controls the first terminal device to perform AR remote interaction with the second terminal device, and the second terminal device obtains the video data to be tested through a second camera; The server determines that the AR scene and the video data to be tested meet the preset consistency condition, including: The server receives the video data to be measured sent by the second terminal device, and obtains the target detection frame through the scene identification number to determine the tracking object from the target object; The server determines whether the object to be detected in the video data to be detected and the tracking object meet a preset matching condition; If the determination result is yes, the server determines whether there is a reference plane in the video data to be tested, so as to determine that the AR scene and the video data to be tested meet the preset consistency condition when it is determined that the reference plane exists, and the preset consistency condition includes the preset matching condition and the reference plane; The server determines whether the object to be detected in the video data to be detected and the tracked object meet the preset matching condition, including: The server runs a preset object detector and obtains an initial bounding box to be measured in an initial frame of the video data to be measured, wherein the initial bounding box to be measured is used to mark the object to be measured in the initial frame; Acquire subsequent bounding boxes to be measured in subsequent frames of the video data to be measured, so as to gradually iterate the initial bounding box to be measured through the subsequent bounding boxes to be measured; Determine the similarity between the target detection box marking the tracked object and each bounding box to be detected, so that when the similarity is less than a preset similarity threshold, end the step-by-step iteration to determine that the tracked object and the object to be detected meet the preset matching condition.

2. The AR remote interaction method according to claim 1, characterized in that: After determining that the AR scene and the video data to be tested meet the preset consistency condition, the method further includes: The server sends an interaction instruction to the second terminal device, so that the second terminal device renders the AR material on the reference plane according to the interaction instruction to obtain a participating AR object, where the participating AR object is used to interact with the initial AR object of the first terminal device.

3. The AR remote interaction method according to claim 1 or 2, characterized in that: After controlling the first terminal device to perform AR remote interaction with the second terminal device, the method further includes: The server receives the respective newly added data packets sent by the first terminal device and / or the second terminal device in real time, and updates the cloud anchor point according to the received newly added data packets; Among them, the first terminal device and / or the second terminal device compares their respective local anchor points with the cloud anchor point according to their respective preset time periods, and when the first terminal device and / or the second terminal device determines that there is a new anchor point, they respectively send modification requests to the server.

4. The AR remote interaction method according to claim 3, characterized in that: After sending the modification request to the server, the method further comprises: The server generates idle state feedback information according to the modification request, and transmits the idle state feedback information back to the first terminal device or the second terminal device, so that the first terminal device or the second terminal device that receives the idle state feedback information modifies the local anchor point.

5. An augmented reality (AR) remote interaction system, characterized in that: include: A first terminal device, a server, and a second terminal device; The first terminal device is used to create an AR scene, and sends a data packet generated by creating the AR scene to a server, wherein the AR scene includes an initial AR object obtained by rendering an AR material, and the data packet includes anchor point coordinates, a size of the AR material, coordinates of the first terminal device, and a target detection frame, wherein the target detection frame is used to mark the target object in the AR scene; the first terminal device also includes a first camera, wherein the first camera is used to obtain environmental video data of a space where the first terminal device is located, and the environmental video data is used to obtain the target detection frame; The server is used to generate a scene identification number according to the data packet sent by the first terminal device, so that the second terminal device participates in the AR scene according to the scene identification number, and the scene identification number is used to uniquely identify the AR scene; The server is further configured to control the first terminal device to perform AR remote interaction with the second terminal device if it is determined that the AR scene and the video data to be tested meet a preset consistency condition; The second terminal device includes a second camera, and the second camera is used to obtain the video data to be tested; Wherein, the server includes: A first processing module, configured to receive the video data to be measured sent by the second terminal device, and obtain the target detection frame through the scene identification number to determine the tracking object from the target object; A second processing module is used to determine whether the object to be detected in the video data to be detected and the tracking object meet a preset matching condition; If the determination result is yes, the second processing module is further used to determine whether there is a reference plane in the video data to be tested, so as to determine that the AR scene and the video data to be tested meet the preset consistency condition when it is determined that the reference plane exists, and the preset consistency condition includes the preset matching condition and the reference plane; Wherein, the second processing module is specifically used for: Running a preset object detector and obtaining an initial bounding box to be measured in an initial frame of the video data to be measured, wherein the initial bounding box to be measured is used to mark the object to be measured in the initial frame; Acquire subsequent bounding boxes to be measured in subsequent frames of the video data to be measured, so as to gradually iterate the initial bounding box to be measured through the subsequent bounding boxes to be measured; Determine the similarity between the target detection box marking the tracked object and each bounding box to be detected, so that when the similarity is less than a preset similarity threshold, end the step-by-step iteration to determine that the tracked object and the object to be detected meet the preset matching condition.

6. The AR remote interaction system according to claim 5, characterized in that: The server also includes: A first sending module, used for sending an interaction instruction to the second terminal device; The second terminal device also includes: The third processing module is used to render the AR material on the reference plane according to the interaction instruction to obtain a participating AR object, where the participating AR object is used to interact with the initial AR object of the first terminal device.

7. The AR remote interaction system according to claim 5 or 6, characterized in that: The first terminal device and / or the second terminal device further includes: A second sending module, configured to send respective newly added data packets to the server in real time, so that the server updates the cloud anchor point according to the received newly added data packets; a fourth processing module, configured to compare respective local anchor points with the cloud anchor points according to respective preset time periods; If it is determined that there is a newly added anchor point, the second sending module is further configured to send a modification request to the server respectively.

8. The AR remote interaction system according to claim 7, characterized in that: The server further comprises: The fifth processing module is used to generate idle state feedback information according to the modification request, and transmit the idle state feedback information back to the first terminal device or the second terminal device, so that the first terminal device or the second terminal device that receives the idle state feedback information modifies the local anchor point.

9. An augmented reality (AR) remote interaction system, characterized in that: include: A first camera and a second camera; processor; as well as a memory for storing a computer program for the processor; Wherein, the processor is configured to perform the AR remote interaction method according to any one of claims 1 to 4 by executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the AR remote interaction method according to any one of claims 1 to 4 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the AR remote interaction method according to any one of claims 1 to 4 is implemented.

Images