A remote annotation method, terminal and system based on mixed reality

Abstract

The present invention proposes a remote annotation method, terminal and system based on mixed reality, which obtain a two-dimensional annotation array; convert the obtained two-dimensional annotation array into a three-dimensional coordinate array; calculate the center of gravity coordinates based on the sum of all coordinates in each direction in the three-dimensional coordinate array; emit a ray at the center of gravity coordinates, determine the object hit by the ray as the object to be annotated, subtract the coordinates of the collision point between the ray and the object from the center of gravity coordinates to obtain a transformation displacement; add each point in the three-dimensional coordinate array to the transformation displacement to obtain a new three-dimensional array; render a line segment between every two points in the new three-dimensional array to complete real-time three-dimensional annotation in space; the method and system proposed by the present invention can realize remote mixed reality annotation between a Hololens terminal and other devices such as a PC terminal, and have high accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of remote annotation technology, and in particular relates to a remote annotation method, terminal and system based on mixed reality. Background Art

[0002] In work and life, when people encounter problems, they may seek remote assistance. However, the objects that need to be operated are often in a relatively complex real environment. If we rely on traditional video interaction methods, we can only view from a one-way perspective, cannot use body language, and find it difficult to describe the environment and location of real objects in words. In addition, using the naked eye to view and describe the location of real objects is prone to errors.

[0003] The inventors found that traditional video interaction methods can only view one-way perspectives, cannot use body language, and find it difficult to describe the environment of real objects and their locations in words. It cannot accurately achieve remote labeling between the assisting end (PC end) and the assisted end (Hololens end), and the accuracy is low. Summary of the Invention

[0004] In order to solve the above problems, the present invention proposes a remote annotation method, terminal and system based on mixed reality. The method proposed in the present invention can realize remote mixed reality annotation between Hololens and other devices such as PC, and has high accuracy.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a remote annotation method based on mixed reality, which adopts the following technical solutions:

[0006] A remote annotation method based on mixed reality, comprising:

[0007] Get the two-dimensional annotation array;

[0008] Convert the obtained two-dimensional annotation array into a three-dimensional coordinate array;

[0009] The center of gravity coordinates are calculated based on the sum of all coordinates in each direction in the three-dimensional coordinate array; a ray is emitted at the center of gravity coordinates, and the object hit by the ray is determined to be the object to be labeled. The coordinates of the collision point between the ray and the object are subtracted from the center of gravity coordinates to obtain the transformation displacement; each point in the three-dimensional coordinate array is added to the transformation displacement to obtain a new three-dimensional array; a line segment is rendered between every two points in the new three-dimensional array to complete real-time three-dimensional labeling in space.

[0010] Furthermore, when obtaining the two-dimensional annotation array, a line renderer and an inverse interpolation algorithm are used for two-dimensional annotation.

[0011] Furthermore, the interpolation lerp between two numbers y1 and y2 is: lerp = y1 + (y2 - y1) * weighy;

[0012] Among them, weight is a real number in the interval [0, 1]. The inverse interpolation algorithm uses the known interpolation value lerp and two numbers y1 and y2 to find the value of weight.

[0013] Furthermore, the screen space coordinates of the mouse in each frame are automatically captured and saved in the storage coordinate array, and a point is added; at the same time, the last two coordinates in the coordinate array are used to form a new coordinate array for each frame, and the last two coordinates are (point1, point2); the loop is repeated multiple times, and each loop uses 0 as y1, 3 as y2, and 0 as interpolation to calculate the value of weight, the supplementary coordinate = (1-weight)*point1+weight*points2, and the supplementary coordinate is added to a new coordinate array, and the interpolation value is increased by 1 to enter the next loop. After the loop is repeated multiple times, the supplementary coordinates between the two points and the original two coordinate points will be obtained. The obtained coordinates are connected between each point to obtain a smooth curve until the assisting end finishes labeling.

[0014] Furthermore, when converting the acquired two-dimensional annotation array into a three-dimensional coordinate array, the two-dimensional coordinate array of the screen is converted into a three-dimensional coordinate array at the same position from the object screen used by the assisted end.

[0015] Among them, x m is the sum of the x-axis and y-axis of all coordinates m is the sum of the y-axis of all coordinates, z m is the sum of the z-axis of all coordinates, and m is the number of coordinates.

[0016] In order to achieve the above objectives, in a second aspect, the present invention further provides a remote annotation terminal based on mixed reality, which adopts the following technical solutions:

[0017] A remote annotation terminal based on mixed reality includes at least a processor, wherein the processor is configured to:

[0018] Get the two-dimensional annotation array;

[0019] Convert the obtained two-dimensional annotation array into a three-dimensional coordinate array;

[0020] The center of gravity coordinates are calculated based on the sum of all coordinates in each direction in the three-dimensional coordinate array; a ray is emitted at the center of gravity coordinates, and the object hit by the ray is determined to be the object to be labeled. The coordinates of the collision point between the ray and the object are subtracted from the center of gravity coordinates to obtain the transformation displacement; each point in the three-dimensional coordinate array is added to the transformation displacement to obtain a new three-dimensional array; a line segment is rendered between every two points in the new three-dimensional array to complete real-time three-dimensional labeling in space.

[0021] Furthermore, the marked terminal is an assisted terminal, which is connected to other assisting terminals through a remote server; when initiating a video voice call invitation, the remote server pulls the token of the marked terminal and the token of the assisted terminal into the same room, and sends each other's Internet Protocol address and port to both parties, so that the assisting terminal and the assisted terminal establish a user datagram protocol.

[0022] In order to achieve the above objectives, in a third aspect, the present invention further provides a remote annotation system based on mixed reality, which adopts the following technical solutions:

[0023] A remote annotation system based on mixed reality, comprising:

[0024] a remote server, and at least one assisting terminal and at least one assisted terminal connected thereto;

[0025] The assisted terminal is a marking terminal, which is connected to other assisting terminals via a remote server. When initiating a video or voice call invitation, the remote server pulls the token of the marking terminal and the token of the assisted terminal into the same room and sends each other's Internet Protocol address and port to establish a User Datagram Protocol (UDP) connection between the assisting terminal and the assisted terminal.

[0026] The annotation module is configured to: calculate the center of gravity coordinates based on the sum of all coordinates in all directions in the three-dimensional coordinate array; emit a ray at the center of gravity coordinates, determine the object hit by the ray as the object to be annotated, subtract the coordinates of the point where the ray and the object collide from the center of gravity coordinates to obtain a transformation displacement; add each point in the three-dimensional coordinate array to the transformation displacement to obtain a new three-dimensional array; render a line segment between every two points in the new three-dimensional array to complete real-time three-dimensional annotation in space.

[0027] In order to achieve the above objectives, in a fourth aspect, the present invention further provides a remote annotation system based on mixed reality, which adopts the following technical solutions:

[0028] A remote annotation system based on mixed reality, including a data acquisition module, a conversion module and an annotation module;

[0029] The data acquisition module is configured to: obtain a two-dimensional annotation array;

[0030] The conversion module is configured to: convert the acquired two-dimensional annotation array into a three-dimensional coordinate array;

[0031] The annotation module is configured to: calculate the center of gravity coordinates based on the sum of all coordinates in all directions in the three-dimensional coordinate array; emit a ray at the center of gravity coordinates, determine the object hit by the ray as the object to be annotated, subtract the coordinates of the point where the ray and the object collide from the center of gravity coordinates to obtain a transformation displacement; add each point in the three-dimensional coordinate array to the transformation displacement to obtain a new three-dimensional array; render a line segment between every two points in the new three-dimensional array to complete real-time three-dimensional annotation in space.

[0032] In order to achieve the above-mentioned objectives, in a fifth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the remote annotation method based on mixed reality described in the first aspect.

[0033] In order to achieve the above-mentioned objectives, in the sixth aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the remote annotation method based on mixed reality described in the first aspect are implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] In the present invention, the received two-dimensional coordinate array is converted into a three-dimensional coordinate array at the same point of the object screen used by the assisted end, the center of gravity coordinates are calculated, and a ray is emitted using the center of gravity coordinates. The object that the ray collides with is the object to be marked, and the coordinates of the collision point and the center of gravity coordinate point are subtracted to obtain the transformation displacement. The transformation displacement is added to each point in the three-dimensional coordinate array to obtain a new three-dimensional array, and a line segment is rendered between every two points in the new three-dimensional array to complete real-time three-dimensional annotation in space. Experimental results show that the method and system proposed in the present invention can realize remote mixed reality annotation between the Hololens end and other devices such as the PC end, and have high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings constituting a part of the specification of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments and descriptions of this embodiment are used to explain this embodiment and do not constitute an improper limitation on this embodiment.

[0037] Figure 1 This is a schematic diagram of the encapsulation of application layer communication in Example 1 of the present invention;

[0038] Figure 2This is a schematic diagram of the specific structure of the voice and video call in Example 1 of the present invention;

[0039] Figure 3 This is a two-dimensional annotation flow chart of Example 1 of the present invention;

[0040] Figure 4 This is a three-dimensional annotation flow chart of Example 1 of the present invention;

[0041] Figure 5 This is a schematic diagram of the labeling effect performed by the assisting terminal according to Example 1 of the present invention;

[0042] Figure 6 This is a schematic diagram of the real-time display of annotation effects by the assisted terminal according to embodiment 1 of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0045] Example 1:

[0046] like Figure 1 As shown, this embodiment provides a remote annotation method based on mixed reality, including:

[0047] Get the two-dimensional annotation array;

[0048] Convert the obtained two-dimensional annotation array into a three-dimensional coordinate array;

[0049] The center of gravity coordinates are calculated based on the sum of all coordinates in each direction in the three-dimensional coordinate array; a ray is emitted at the center of gravity coordinates, and the object hit by the ray is determined to be the object to be labeled. The coordinates of the collision point between the ray and the object are subtracted from the center of gravity coordinates to obtain the transformation displacement; each point in the three-dimensional coordinate array is added to the transformation displacement to obtain a new three-dimensional array; a line segment is rendered between every two points in the new three-dimensional array to complete real-time three-dimensional labeling in space.

[0050] The method in this embodiment can be implemented based on a remote server, an assisting terminal (i.e., the assisting end) and a labeling terminal (i.e., the assisted end), and the server is connected to the assisting end and the assisted end; in this embodiment, the assisting end is configured to: perform two-dimensional labeling and send the two-dimensional labeling to the assisted end; the assisted end is configured to: convert the two-dimensional labeling into three-dimensional labeling, and display the specific position in the real space in real time; convert the received two-dimensional coordinate array into a three-dimensional coordinate array at the same point of the object screen used by the assisted end, calculate the center of gravity coordinates, and use the center of gravity coordinates to emit a ray. The object that the ray collides with is the object to be labeled, and the collision point coordinates and the center of gravity coordinate point are subtracted to obtain the transformation displacement, and the transformation displacement is added to each point in the three-dimensional coordinate array to obtain a new three-dimensional array, and a line segment is rendered between every two points of the new three-dimensional array to complete real-time three-dimensional labeling in space.

[0051] Specifically, such as Figure 2 As shown, the working process or principle of the remote annotation system based on mixed reality in this embodiment is as follows:

[0052] S1. Build a remote server and connect the assisted end and the assisting end;

[0053] In the process of building the server, the Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) are combined to form a method. Because the TCP protocol has very good reliability, before using TCP to transmit data, in order to track and negotiate each data sent to ensure the synchronization of sending and receiving at the data end, there will be a three-way handshake to establish a connection. In addition, there will be confirmation responses, timeout retransmission, sliding windows, congestion control, flow control and delayed response mechanisms to ensure data integrity during data transmission. After the data transmission is completed, the connection will be automatically disconnected to save system resources. The TCP protocol is used for user-oriented connections. Because the UDP protocol does not have confirmation responses, timeout retransmission, sliding windows, congestion control, flow control and delayed response mechanisms, it is a stateless transmission protocol, so it has very low delay during transmission. In order to make the video low-latency and smooth real-time transmission, the UDP protocol is used to establish a point-to-point connection between the assisting end and the assisted end, which greatly reduces the delay.

[0054] like Figure 1As shown in the figure, the method of establishing a user pool (UserTokenPool) is used to realize multi-person video communication by pulling users into the room (Room). The server uses the UnityNetWorkManager class to uniformly handle the establishment and disconnection requests of the two protocols. The server passes the received information to the MessageHandlerCenter class for processing, identifying the type and content of the information and performing corresponding operations. The server uses the MessageSendManager class to send the request, and then the application layer is only responsible for interacting with the MessageHandlerCenter class and MessageSendManager class to realize the encapsulation of application layer communication.

[0055] When the server is started, it initializes the port and user pool, registers for client connection events, and registers for receive and send events for all users in the pool. Each time a user connects to the server, a user token is popped from the pool and assigned the current connected user's socket. When the server receives a user's request, it makes a response and returns a result. The server checks the heartbeat interval. Users send heartbeat packets to the server every 30 seconds to determine if they are online. If a user times out, the user is disconnected, or the user actively disconnects.

[0056] S2. Establish a video connection;

[0057] After the assisting end and the assisted end establish a connection with the server, when the assisting end and the assisted end initiate a video or voice call invitation, the remote server will pull the assisting end's token and the assisted end's token into the same room and send each other's Internet Protocol address (IP) and port to establish a connection between the assisting end and the assisted end.

[0058] User Datagram Protocol (UDP) connection.

[0059] After establishing a connection, use the Unity application programming interface (API) to open the specified webcam to obtain video data and create a texture (WebCam Texture) for real-time video input rendering. The texture rendered by the camera in real time needs to be converted into Texture 2D format in real time to obtain video data. Use RenderTexture as an intermediary to create Texture2D by directly pointing to the memory address of the texture rendered by the camera in real time. Create an image and save it in the texture data, saving memory to complete the conversion to obtain video data.

[0060] S3, the assisting end performs annotation and sends the two-dimensional annotation to the assisted end;

[0061] like Figure 3 As shown, the line renderer and inverse interpolation algorithm can be used in the annotation process, specifically:

[0062] The interpolation lerp between two numbers y1 and y2 is lerp = y1 + (y2 - y1) * weigt, where weigt is a real number in the interval [0, 1]. The inverse interpolation algorithm uses the known interpolation lerp and the two numbers y1 and y2 to find the value of weight.

[0063] After the assisting client clicks to start labeling, each time the left mouse button is held down, the screen-space coordinates of the mouse are automatically captured and saved in the coordinate storage array (Positions), and a point is added. At the same time, each frame, the last two coordinates (point1, point2) in the Positions array are used to form a new coordinate array (NewPositions). This loop is repeated multiple times, four times in this embodiment. In each loop, 0 is used as y1, 3 as y2, and 0 as the lerp value. The value of weigt is calculated using an algorithm to obtain the supplementary coordinates = (1-weigt)*point1+weigt*points2. The calculated supplementary coordinates are added to the new coordinate array, and the lerp value is incremented by 1 to enter the next loop. After four loops, the supplementary coordinates between the two points and the original two coordinates are obtained, as shown in Table 1. The obtained coordinates are connected between each point to form a smooth curve. Until the assisting client finishes labeling, the assisting client sends the Positions array to the assisted client.

[0064] Table 1 Supplementary coordinates and the original two coordinate points

[0065]

[0066] S4, the assisted end converts the two-dimensional annotation into a three-dimensional annotation, and displays the specific location in the display space in real time, thereby completing the annotation;

[0067] like Figure 4 As shown, after the assisted end receives the coordinate array from the assisting end, the received coordinates are converted into a three-dimensional coordinate array at the same distance from the assisted end user (camera) screen using the Unity application programming interface (API). Among them, x m is the sum of the x-axis and y-axis of all coordinates m is the sum of the y-axis of all coordinates, z mis the sum of the z-axis of all coordinates, and m is the number of coordinates; calculate the centroid coordinates (x, y, z), use the centroid coordinates to send a ray forward, the object that the ray collides with is the object to be labeled, and save the coordinates of the collision point (n, o, a), subtract the collision point coordinates from the centroid coordinate point to obtain the transformation displacement (a, b, c), add the transformation displacement (a, b, c) to each point in the three-dimensional coordinate array to obtain a new three-dimensional array, render a line segment between every two points in the new three-dimensional array, and complete real-time three-dimensional labeling in space.

[0068] S5, such as Figure 5 and Figure 6 As shown, the experimental results show that the method proposed in this patent can realize remote labeling between the assisting and assisted devices with relatively high accuracy.

[0069] Example 2:

[0070] This embodiment provides a remote annotation terminal based on mixed reality, which includes at least a processor. Other configurations of the terminal are conventional technologies and will not be described in detail here. The processor is configured to:

[0071] Get the two-dimensional annotation array;

[0072] Convert the obtained two-dimensional annotation array into a three-dimensional coordinate array;

[0073] The center of gravity coordinates are calculated based on the sum of all coordinates in each direction in the three-dimensional coordinate array; a ray is emitted at the center of gravity coordinates, and the object hit by the ray is determined to be the object to be labeled. The coordinates of the collision point between the ray and the object are subtracted from the center of gravity coordinates to obtain the transformation displacement; each point in the three-dimensional coordinate array is added to the transformation displacement to obtain a new three-dimensional array; a line segment is rendered between every two points in the new three-dimensional array to complete real-time three-dimensional labeling in space.

[0074] Furthermore, the marked terminal is an assisted terminal, which is connected to other assisting terminals through a remote server; when initiating a video voice call invitation, the remote server pulls the token of the marked terminal and the token of the assisted terminal into the same room, and sends each other's Internet Protocol address and port to both parties, so that the assisting terminal and the assisted terminal establish a user datagram protocol.

[0075] The working method of the system is the same as the remote annotation method based on mixed reality in Example 1, and will not be repeated here.

[0076] Example 3:

[0077] A remote annotation system based on mixed reality, comprising:

[0078] a remote server, and at least one assisting terminal and at least one assisted terminal connected thereto;

[0079] The assisted terminal is a marking terminal, which is connected to other assisting terminals via a remote server. When initiating a video or voice call invitation, the remote server pulls the token of the marking terminal and the token of the assisted terminal into the same room and sends each other's Internet Protocol address and port to establish a User Datagram Protocol (UDP) connection between the assisting terminal and the assisted terminal.

[0080] The annotation module is configured to: calculate the center of gravity coordinates based on the sum of all coordinates in all directions in the three-dimensional coordinate array; emit a ray at the center of gravity coordinates, determine the object hit by the ray as the object to be annotated, subtract the coordinates of the point where the ray and the object collide from the center of gravity coordinates to obtain a transformation displacement; add each point in the three-dimensional coordinate array to the transformation displacement to obtain a new three-dimensional array; render a line segment between every two points in the new three-dimensional array to complete real-time three-dimensional annotation in space.

[0081] The working method of the system is the same as the remote annotation method based on mixed reality in Example 1, and will not be repeated here.

[0082] Example 4:

[0083] This embodiment provides a remote annotation system based on mixed reality, including a data acquisition module, a conversion module and an annotation module;

[0084] The data acquisition module is configured to: obtain a two-dimensional annotation array;

[0085] The conversion module is configured to: convert the acquired two-dimensional annotation array into a three-dimensional coordinate array;

[0086] The annotation module is configured to: calculate the center of gravity coordinates based on the sum of all coordinates in all directions in the three-dimensional coordinate array; emit a ray at the center of gravity coordinates, determine the object hit by the ray as the object to be annotated, subtract the coordinates of the point where the ray and the object collide from the center of gravity coordinates to obtain a transformation displacement; add each point in the three-dimensional coordinate array to the transformation displacement to obtain a new three-dimensional array; render a line segment between every two points in the new three-dimensional array to complete real-time three-dimensional annotation in space.

[0087] The working method of the system is the same as the remote annotation method based on mixed reality in Example 1, and will not be repeated here.

[0088] Example 5:

[0089] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the remote annotation method based on mixed reality described in Example 1 are implemented.

[0090] Example 6:

[0091] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the remote annotation method based on mixed reality described in Example 1 are implemented.

[0092] The above description is merely a preferred embodiment of this embodiment and is not intended to limit this embodiment. Those skilled in the art will readily appreciate that this embodiment may be modified and varied in various ways. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this embodiment shall be within the scope of protection of this embodiment.

Claims

1. A remote annotation method based on mixed reality, characterized in that: include: Get the two-dimensional annotation array; Convert the obtained two-dimensional annotation array into a three-dimensional coordinate array; Utilizing algorithms , find the coordinates of the center of gravity; where, For all coordinates The sum of the axes, For all coordinates The sum of the axes, For all coordinates The sum of the axes, is the number of coordinates; a ray is emitted at the barycentric coordinate, and the object hit by the ray is determined to be the object to be annotated. The coordinates of the point where the ray and the object collide are subtracted from the barycentric coordinate to obtain a transformation displacement; each point in the three-dimensional coordinate array is added to the transformation displacement to obtain a new three-dimensional array; a line segment is rendered between every two points in the new three-dimensional array to complete real-time three-dimensional annotation in space; The method of obtaining a two-dimensional annotation array and converting the obtained two-dimensional annotation array into a three-dimensional coordinate array can be implemented based on a remote server, an assisting terminal and an assisted terminal, wherein the server is connected to the assisting terminal and the assisted terminal; The assisting end is configured to: perform two-dimensional annotation and send the two-dimensional annotation to the assisted end; the assisted end is configured to: convert the two-dimensional annotation array into a three-dimensional array and display the specific position in the real space in real time.

2. The remote annotation method based on mixed reality according to claim 1, characterized in that: When obtaining a 2D annotation array, a line renderer and an inverse interpolation algorithm are used for 2D annotation.

3. The remote annotation method based on mixed reality according to claim 2, characterized in that: Two numbers Interpolation between for: = +( - )* ; in, Is a real number in the interval [0, 1], the inverse interpolation algorithm uses the known interpolation and two numbers Find Value .

4. The remote annotation method based on mixed reality according to claim 3, characterized in that: The screen space coordinates of the mouse are automatically captured and saved in the storage coordinate array for each frame, and a point is added; at the same time, the last two coordinates in the coordinate array are used to form a new coordinate array for each frame. The last two coordinates are ( point1, point2 ); loop multiple times, each loop uses 0 as , 3 as , 0 as interpolation, find The value of the supplementary coordinates = (1- )* point1 + * points2 , add the supplementary coordinates to a new coordinate array, and add 1 to the interpolation value to enter the next loop. After multiple cycles, the supplementary coordinates between the two points and the original two coordinate points will be obtained. Connect each point of the obtained coordinates to obtain a smooth curve until the assisting end finishes labeling.

5. A remote annotation terminal based on mixed reality using the remote annotation method based on mixed reality according to claim 1, characterized in that: At least a processor is included, wherein the processor is configured to: Get the two-dimensional annotation array; Convert the obtained two-dimensional annotation array into a three-dimensional coordinate array; Utilizing algorithms , find the coordinates of the center of gravity; where, For all coordinates The sum of the axes, For all coordinates The sum of the axes, For all coordinates The sum of the axes, is the number of coordinates; a ray is emitted at the barycentric coordinate, and the object hit by the ray is determined to be the object to be labeled, and the coordinates of the collision point between the ray and the object are subtracted from the barycentric coordinate to obtain the transformation displacement; each point in the three-dimensional coordinate array is added to the transformation displacement to obtain a new three-dimensional array; a line segment is rendered between every two points in the new three-dimensional array to complete real-time three-dimensional labeling in space.

6. A remote annotation system based on mixed reality using the remote annotation method based on mixed reality according to claim 1, characterized in that: include: a remote server, and at least one assisting terminal and at least one assisted terminal connected thereto; The assisted terminal is a marking terminal, which is connected to other assisting terminals via a remote server. When initiating a video or voice call invitation, the remote server pulls the token of the marking terminal and the token of the assisted terminal into the same room and sends each other's Internet Protocol address and port to establish a User Datagram Protocol (UDP) connection between the assisting terminal and the assisted terminal. The annotation module is configured to: calculate the center of gravity coordinates based on the sum of all coordinates in all directions in the three-dimensional coordinate array; emit a ray at the center of gravity coordinates, determine the object hit by the ray as the object to be annotated, subtract the coordinates of the point where the ray and the object collide from the center of gravity coordinates to obtain a transformation displacement; add each point in the three-dimensional coordinate array to the transformation displacement to obtain a new three-dimensional array; render a line segment between every two points in the new three-dimensional array to complete real-time three-dimensional annotation in space.

7. A remote annotation system based on mixed reality using the remote annotation method based on mixed reality according to claim 1, characterized in that: Including data acquisition module, conversion module and annotation module; The data acquisition module is configured to: obtain a two-dimensional annotation array; The conversion module is configured to: convert the acquired two-dimensional annotation array into a three-dimensional coordinate array; The annotation module is configured to: calculate the center of gravity coordinates based on the sum of all coordinates in all directions in the three-dimensional coordinate array; emit a ray at the center of gravity coordinates, determine the object hit by the ray as the object to be annotated, subtract the coordinates of the point where the ray and the object collide from the center of gravity coordinates to obtain a transformation displacement; add each point in the three-dimensional coordinate array to the transformation displacement to obtain a new three-dimensional array; render a line segment between every two points in the new three-dimensional array to complete real-time three-dimensional annotation in space.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the remote annotation method based on mixed reality are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the remote annotation method based on mixed reality are implemented as described in any one of claims 1 to 4.

Images