Synchronization method and device of marking information, computer device and remote collaboration system

Abstract

The application belongs to the technical field of computer vision, and particularly relates to a marker information synchronization method and device, a computer device and a remote collaboration system. The method comprises the following steps: obtaining marker information of a target terminal device and a 2D coordinate point set of the marker information on a scene image; obtaining a minimum circumscribed rectangle of the 2D coordinate point set; obtaining a mapping background image and a mapping coordinate point set based on the scene image, the 2D coordinate point set and the minimum circumscribed rectangle; obtaining a real-time scene image of all terminal devices once every preset time length; obtaining a homography matrix of an initial background image and each real-time scene image; obtaining a real-time 2D coordinate point set corresponding to each terminal device based on the homography matrix and the mapping coordinate point set; and rendering each real-time 2D coordinate point set into a continuous curve and displaying the continuous curve on a real-time scene image of each terminal device. The application can improve the accuracy of marker information synchronization in a remote collaboration process.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology. Specifically, this application relates to a method, apparatus, computer equipment, and remote collaboration system for synchronizing labeled information. Background Technology

[0002] Remote collaboration breaks down geographical barriers, providing distributed users with electronic shared workspaces. It can be used for real-time AR annotation, multi-view video communication, document delivery, after-sales maintenance, and remote training. Existing remote collaboration systems typically only allow documents, audio, and video created on a local client to be sent to a remote client over the network. In some application scenarios (such as remote teaching and remote conferencing), it is also necessary to annotate target objects in the remote view for purposes such as reminders and emphasis.

[0003] However, the existing technical solutions have poor synchronization accuracy of annotation information, which means that for remote client users, the annotation information they receive cannot truly express the annotation intent of local client users, thus failing to achieve fine-grained interaction between users. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, computer device, and storage medium for synchronizing labeled information, aiming to solve the technical problem of poor accuracy in synchronizing labeled information during remote collaboration.

[0005] To achieve the aforementioned objective, this application provides a method for synchronizing tag information in a remote collaboration system. The remote collaboration system includes at least two terminal devices, each terminal device comprising a display unit and an input unit. The method includes:

[0006] Acquire marking information of the target terminal device, wherein the marking information is displayed in the form of handwriting on the scene image of the display unit of the target terminal device, and the marking information is obtained based on the input unit;

[0007] Obtain the set of 2D coordinate points of the marked information on the scene image;

[0008] Obtain the minimum bounding rectangle of the 2D coordinate point set, and set the image portion of the scene image within the minimum bounding rectangle as the initial background image;

[0009] Based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle, a mapped background image and a set of mapped coordinate points are obtained. The image content of the mapped background image is the same as the image content of the initial background image, and the relative position of the set of mapped coordinate points and the mapped background image is the same as the relative position of the set of 2D coordinate points and the initial background image.

[0010] Every preset time interval, a real-time scene image of the display units of all the terminal devices is acquired once;

[0011] Obtain the homography matrix H of the initial background image and each of the real-time scene images, wherein the homography matrix H makes any pixel coordinate x in the mapped background image and the corresponding pixel coordinate x′ in the real-time scene image satisfy the relationship x′=Hx;

[0012] Based on the homography matrix H and the set of mapped coordinate points, obtain the set of real-time 2D coordinate points corresponding to each terminal device;

[0013] Each set of real-time 2D coordinate points is rendered as a continuous curve and displayed on the real-time scene image of the display unit of the corresponding terminal device, thus completing the synchronization of the marking information.

[0014] In some embodiments, the step of obtaining the set of 2D coordinate points of the marker information on the scene image includes:

[0015] Get the preset number of coordinate points;

[0016] Based on the number of coordinate points, points are taken at equal or unequal intervals on the marking information and saved to obtain a set of 2D coordinate points of the marking information on the scene image.

[0017] In some embodiments, the step of obtaining the mapped background image and the mapped coordinate point set based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle includes:

[0018] The scene image is mapped using transformation matrix A to obtain the mapped background image;

[0019] The transformation matrix A is used to map the 2D coordinate point set to obtain the mapped coordinate point set;

[0020] The transformation matrix A is obtained based on the following formula:

[0021]

[0022] Where w is the width of the minimum bounding rectangle, h is the height of the minimum bounding rectangle, θ is the rotation angle of the minimum bounding rectangle, and cx Let x be the x-value of the center coordinate of the minimum bounding rectangle, and c be the x-value of the center coordinate of the minimum bounding rectangle. y Let y be the center coordinate of the minimum bounding rectangle.

[0023] This application also provides a device for synchronizing marker information in a remote collaboration system, the remote collaboration system including at least two terminal devices, each terminal device including a display unit and an input unit, the device comprising:

[0024] Acquire marking information of the target terminal device, wherein the marking information is displayed in the form of handwriting on the scene image of the display unit of the target terminal device, and the marking information is obtained based on the input unit;

[0025] Obtain the set of 2D coordinate points of the marked information on the scene image;

[0026] Obtain the minimum bounding rectangle of the 2D coordinate point set, and set the image portion of the scene image within the minimum bounding rectangle as the initial background image;

[0027] This is used to obtain a mapped background image and a mapped coordinate point set based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle, wherein the image content of the mapped background image is the same as the image content of the initial background image, and the relative position of the mapped coordinate point set and the mapped background image is the same as the relative position of the set of 2D coordinate points and the initial background image.

[0028] Every preset time interval, a real-time scene image of the display units of all the terminal devices is acquired once;

[0029] Used to obtain the homography matrix H between the mapped background image and each of the real-time scene images, wherein the homography matrix H makes any pixel coordinate x in the mapped background image and the corresponding pixel coordinate x′ in the real-time scene image satisfy the relationship x′=Hx;

[0030] Based on the homography matrix H and the set of mapped coordinate points, obtain the set of real-time 2D coordinate points corresponding to each terminal device;

[0031] Each set of 2D coordinate points is rendered as a continuous curve and displayed on the scene image of the display unit of each terminal device, thus completing the synchronization of the marking information.

[0032] This application also provides a remote collaboration system, including: at least two terminal devices and a controller, wherein the terminal devices include a display unit and an input unit, and the controller is used to execute the steps of the synchronization method for tag information provided in any of the above embodiments.

[0033] This application also provides a remote collaboration system, including: at least two terminal devices, each terminal device including a display unit, an input unit, a sending unit, a receiving unit, and a control and calculation unit, wherein the at least two terminal devices include a receiving terminal device and a sending terminal device;

[0034] In the sending terminal device, when the control calculation unit receives the tag information confirmation command issued by the input unit, it performs the following steps:

[0035] Obtain the marking information displayed in the scene image in the display unit;

[0036] Obtain the set of 2D coordinate points of the marked information on the scene image;

[0037] Obtain the minimum bounding rectangle of the 2D coordinate point set, and set the image portion of the scene image within the minimum bounding rectangle as the initial background image;

[0038] Based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle, a mapped background image and a set of mapped coordinate points are obtained. The image content of the mapped background image is the same as the image content of the initial background image, and the relative position of the set of mapped coordinate points and the mapped background image is the same as the relative position of the set of 2D coordinate points and the initial background image.

[0039] The mapped background image and the mapped coordinate point set are bound together to obtain a reference dataset.

[0040] The reference dataset is sent to all the receiving terminal devices through the sending unit;

[0041] After a first preset time interval, the first real-time scene image of the display unit is acquired once;

[0042] Obtain the first homography matrix H1 between the mapped background image and the first real-time scene image, wherein the homography matrix H1 such that any pixel coordinate x1 in the mapped background image and the corresponding pixel coordinate x1′ in the first real-time scene image satisfy x1 ′ =H1*x1 relationship;

[0043] Based on the homography matrix H1 and the set of mapped coordinate points, a first real-time 2D coordinate point set is obtained;

[0044] The first set of real-time 2D coordinate points is rendered as a continuous curve and displayed on the scene image of the display unit;

[0045] In the receiving terminal device, when the receiving unit receives the reference dataset, the control calculation unit performs the following steps:

[0046] Save the reference dataset;

[0047] Every second preset time interval, the second real-time scene image of the display unit is acquired once;

[0048] Obtain the second homography matrix H2 between the mapped background image and the second real-time scene image, wherein the homography matrix H2 such that any pixel coordinate x1 in the mapped background image and the corresponding pixel coordinate x2′ in the first real-time scene image satisfy x2 ′ =H2*x1 relationship;

[0049] Based on the second homography matrix H2 and the set of mapped coordinate points, obtain the second 2D real-time coordinate point set;

[0050] The set of 2D real-time coordinate points is rendered as a continuous curve and displayed on the scene image of the display unit.

[0051] In some embodiments, the terminal device further includes a camera, which is used to acquire real-time scene images and transmit them to the display unit for display.

[0052] In some embodiments, the remote collaboration system further includes a server;

[0053] The server is used to receive and store reference datasets from the sending terminal device, and to send the reference datasets to the receiving terminal device.

[0054] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the method for synchronizing tag information provided in any of the above embodiments.

[0055] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for synchronizing tag information provided in any of the above embodiments.

[0056] Beneficial effects:

[0057] This application provides a method, apparatus, computer device, and remote collaboration system for synchronizing marker information. By continuously acquiring real-time scene images in the real space, and mapping a mapped background image and a set of mapped coordinate points based on the marker information determined in the target terminal device and the position of the marker information on the currently displayed scene image of the target terminal device, a homography matrix is ​​obtained between the mapped background image and the real-time scene image. This allows for the localization of a real-time 2D coordinate point set based on the mapped coordinate point set and the homography matrix. Finally, the real-time 2D coordinate point set of each terminal device is rendered as a continuous curve and displayed on the real-time scene image of the corresponding display unit, thus completing the synchronization of marker information. Because the marker information deforms according to its actual positioning result in the real-time scene image, the updated marker information still fits well with the background texture, thereby improving the accuracy of marker information synchronization during remote collaboration. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating a method for synchronizing tag information according to an embodiment of this application;

[0059] Figure 2 This is a schematic diagram of the marking information in one embodiment of this application;

[0060] Figure 3 This is a schematic diagram of the minimum bounding rectangle in one embodiment of this application;

[0061] Figure 4 This is a schematic diagram of a mapped background image and a set of mapped coordinate points in one embodiment of this application;

[0062] Figure 5 This is a schematic diagram illustrating the relationship between the camera coordinate system and the world coordinate system.

[0063] Figure 6 for Figure 2 This is a schematic diagram illustrating the mapping between the template image and the target image.

[0064] Figure 7 This is a flowchart illustrating step S102 in a method for synchronizing tag information according to another embodiment of this application.

[0065] Figure 8 This is a flowchart illustrating step S104 of a method for synchronizing tag information according to an embodiment of this application.

[0066] Figure 9 This is a schematic diagram of the structure of a tag information synchronization device according to an embodiment of this application;

[0067] Figure 10 This is a flowchart illustrating the execution steps of the control motion unit of the transmitting terminal device in one embodiment of this application;

[0068] Figure 11 This is a flowchart illustrating the execution steps of the control motion unit of the receiving terminal device in one embodiment of this application.

[0069] Figure 12 This is a schematic diagram of the structure of a remote collaboration system according to an embodiment of this application;

[0070] Figure 13 This is a schematic diagram of the structure of a computer device according to an embodiment of this application.

[0071] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0073] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.

[0074] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0075] Please refer to Figure 1 This application provides a method for synchronizing tag information in a remote collaboration system. The remote collaboration system includes at least two terminal devices, each including a display unit and an input unit. The method executes steps S101-S108, which are described in detail below:

[0076] S101. Obtain the marking information of the target terminal device, wherein the marking information is displayed in the form of handwriting on the scene image of the display unit of the target terminal device, and the marking information is obtained based on the input unit.

[0077] Specifically, users input marking information on the target terminal device through an input unit, such as a keyboard, mouse, stylus, user's hand and gesture tracking system, touchscreen, or touch surface. The marking information is displayed on the scene image on the display unit of the target terminal device in the form of handwriting drawn by the input unit. It should be noted that all terminal devices in the remote collaboration system can be set as target terminal devices. That is, in the remote collaboration system, new marking information can be added through any terminal device and shared to all other terminal devices. The scene image displayed on the terminal device can be a real-time dynamic image captured by its own camera or an external camera, or a pre-stored or online-shared static image. Furthermore, the objects displayed in the image can be people, animals, objects, text, etc., without limitation.

[0078] For example, please refer to Figure 2 , Figure 2 The illustration shows one form of marking information. In this embodiment, the marking information is represented as an irregular arc-shaped line. In other embodiments, the marking information can also be a straight line, rectangle, ellipse, circle, or other regular or irregular graphic form, which is not limited here.

[0079] S102. Obtain the set of 2D coordinate points of the marker information on the scene image.

[0080] For example, please refer to again Figure 2 ,exist Figure 2 In the illustrated embodiment, the marker information is defined as a set of 2D coordinate points, i.e., the set of 2D coordinate points P = {p0, p1, ..., p2}. n-1}, where p = [x, y, 1] T x is the horizontal coordinate of scene image I in the target terminal device, y is the horizontal and vertical coordinates of scene image I in the target terminal device, and n is the number of coordinate points.

[0081] S103. Obtain the minimum bounding rectangle of the 2D coordinate point set, and set the image portion of the scene image within the minimum bounding rectangle as the initial background image.

[0082] For example, please refer to Figure 3 , Figure 3 This illustrates a representation of a minimum bounding rectangle, in Figure 3 In the embodiment shown, the number of coordinate points is 8, and their positions are as follows: Figure 3 The dots on the marker information indicate that the width of the obtained minimum bounding rectangle is w, the height is h, and the center point coordinates are c = [c x c y ,1] T The rotation angle is θ. In this embodiment, the minimum bounding rectangle of the 2D coordinate point set is obtained using the minAreaRect function of the open-source library OpenCV. Then, the image portion within the minimum bounding rectangle is set as the initial background image.

[0083] S104. Based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle, obtain a mapped background image and a set of mapped coordinate points, wherein the image content of the mapped background image is the same as the image content of the initial background image, and the relative position of the set of mapped coordinate points and the mapped background image is the same as the relative position of the set of 2D coordinate points and the initial background image.

[0084] For example, please refer to Figure 4 , Figure 4 This paper illustrates a representation of a mapped background image and a mapped set of coordinate points. The underlying principle is as follows: For a 2D coordinate point set P and a scene image I, the operation warp(I, A, w, h) = I′ is defined. This maps the coordinate x of each pixel in the scene image I according to the transformation matrix A, resulting in a new pixel coordinate x′ = Ax, forming a mapped background image I′ with width w and height h. The operation warp(P, A) = P′ is also defined. This maps the coordinates of each pixel in the 2D coordinate point set P = {p0, p1, ..., p...}. n-1 Each coordinate point in the matrix is ​​mapped to p′=Ap according to the transformation matrix A, resulting in a new set of points P′={p′0,p′1,..p′ n-1 It should be noted that, in this embodiment, the image content of the mapped background image I′ is the same as the image content of the initial background image (i.e., the image portion of the scene image within the aforementioned minimum bounding rectangle region), and the relative positions of the mapped coordinate point set P′ and the mapped background image I′ are the same as the relative positions of the 2D coordinate point set P and the initial background image.

[0085] S105. After a preset time interval, acquire real-time scene images of the display units of all the terminal devices once.

[0086] To achieve synchronization and tracking of the tagged information across various terminal devices and thus improve its accuracy, the tagged information needs to be updated according to changes in the scene image displayed on each terminal device. The preset duration can be selected based on the human visual persistence effect (approximately 0.1 to 0.4 seconds). When the preset duration (i.e., the update interval) is less than the duration of visual persistence, the update of the tagged information is displayed continuously and smoothly on the display unit, reducing the sense of pause in the image.

[0087] S106. Obtain the homography matrix H of the initial background image and each of the real-time scene images, wherein the homography matrix H makes any pixel coordinate x in the mapped background image and the corresponding pixel coordinate x′ in the real-time scene image satisfy the relationship x′=Hx.

[0088] For details, please refer to Figure 5 Suppose there exists a three-dimensional point X = [X, Y, 1] on a plane in the world coordinate system. T Its projected pixel coordinates in image I are x = [u, v, 1] T There is a relationship between them:

[0089] sx=K[r1,r2,t]X (1)

[0090] Where K is a 3×3 camera intrinsic parameter matrix, R and t are the 3×3 rotation matrix and 3×1 translation vector from the world coordinate system to the camera coordinate system, respectively. s is the scale parameter, which can be any value.

[0091] Define R = [r1, r2, r3]. i Let i ∈ {1, 2, 3} represent the i-th column of the rotation matrix. Then the homography matrix H is:

[0092] H=K[r1, r2, t]=[Kr1, Kr2, Kt]=[h1, h2, h3] (2)

[0093] H is a 3×3 matrix, h i Let be the i-th column of the matrix. Substituting formula (2) into formula (1), we get:

[0094] sx = HX

[0095] The homography matrix H maps a 3D point X on a plane to a 2D point x in an image. Since the 3D point X = [X, Y, 1] on the plane... T If one dimension is fixed (Z=1), then X can be considered as a point on the XY two-dimensional plane. Therefore, the homography matrix H can also be considered as being able to map a point X on the two-dimensional plane to a two-dimensional point x in the image, that is, a two-dimensional to two-dimensional point-to-point mapping.

[0096] Please refer to Figure 6 When a template image and a target image exist, and the goal is to determine the position of the template image in the target image, let the coordinates of any pixel on the template image be x′, and the corresponding pixel coordinates on the target image be x, with the relationship sx = Hx′. Then, the homography matrix H can be used as a representation of the positioning result of the template image on the target image. Therefore, in this application, the homography matrix H between the mapped background image and the real-time scene image of each terminal device can be obtained, i.e., the positional relationship between the two can be obtained. In this embodiment, the mapped background image and the current scene image can be used as input, and the homography matrix H between them can be obtained based on feature point matching methods such as SIFT / SURF / ORB, template matching methods (such as KLT, ESM), or deep learning methods (such as HomographyNet, DELF, etc.).

[0097] S107. Based on the homography matrix H and the set of mapped coordinate points, obtain the set of real-time 2D coordinate points corresponding to each terminal device.

[0098] Since the relative positions of the mapped coordinate point set and the mapped background image are fixed, the positional relationship of the mapped coordinate point set in the real-time scene image can be obtained through the homography matrix H obtained above, thus obtaining the real-time 2D coordinate point set.

[0099] S108. Render each set of real-time 2D coordinate points into a continuous curve and display it on the scene image of the display unit of each terminal device, thereby completing the synchronization of the marking information.

[0100] After obtaining the set of real-time 2D coordinate points, they are rendered into a continuous curve and displayed on the real-time scene image of the display unit of each terminal device, thereby completing the synchronization of the marking information.

[0101] The method for synchronizing marker information provided in this application continuously acquires real-time scene images in the real space. Based on the marker information determined in the target terminal device and the position of the marker information on the scene image currently displayed on the target terminal device, a mapped background image and a set of mapped coordinate points are obtained. A homography matrix between the mapped background image and the real-time scene image is then obtained. Based on the mapping coordinate point set and the homography matrix, a real-time 2D coordinate point set is obtained. Finally, the real-time 2D coordinate point set of each terminal device is rendered as a continuous curve and displayed on the real-time scene image of the corresponding display unit, thus completing the synchronization of marker information. Because the marker information deforms according to its actual positioning result in the real-time scene image, the updated marker information still fits well with the background texture, thereby improving the accuracy of marker information synchronization during remote collaboration.

[0102] In some embodiments, please refer to Figure 7 The step of providing the marked information as a set of 2D coordinate points on the scene image includes:

[0103] S102a, Obtain the preset number of coordinate points.

[0104] Specifically, the appropriate number of coordinate points can be selected according to the actual design requirements. The more coordinate points selected, the higher the accuracy of the marker information synchronization. Conversely, the fewer coordinate points selected, the more distortion may occur during marker information synchronization.

[0105] S102b: Based on the number of coordinate points, take points at equal or unequal intervals on the marking information and save them to obtain a set of 2D coordinate points of the marking information on the scene image.

[0106] Once the number of coordinate points is determined, the points can be arranged according to the specific representation of the marking information. For example, fewer points can be arranged in the straight line segment, while more points can be arranged in the curve or other deformed locations. The positions of the coordinate points can also be selected as equidistant or unequally spaced according to actual needs.

[0107] In some embodiments, please refer to Figure 8 The step of obtaining the mapped background image and the mapped coordinate point set based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle includes:

[0108] S104a. The scene image is mapped using transformation matrix A to obtain a mapped background image;

[0109] S104b. Use the transformation matrix A to map the 2D coordinate point set to obtain the mapped coordinate point set.

[0110] The transformation matrix A is obtained based on the following formula:

[0111]

[0112] Where w is the width of the minimum bounding rectangle, h is the height of the minimum bounding rectangle, θ is the rotation angle of the minimum bounding rectangle, and c x c is the x-value of the center coordinate of the smallest bounding rectangle. y The y-value is the center coordinate of the smallest bounding rectangle.

[0113] By transforming matrix A, even if the minimum bounding rectangle obtained from the set of 2D coordinate points has a certain rotation angle, a normalized mapped background image can still be obtained, which makes subsequent data processing simpler, saves computing resources, and improves computational efficiency.

[0114] Please refer to Figure 9 This application also provides a device for synchronizing tag information in a remote collaboration system. The remote collaboration system includes at least two terminal devices, each terminal device including a display unit and an input unit. The device includes:

[0115] The marker information acquisition module 901 is used to acquire marker information of the target terminal device, wherein the marker information is displayed in the form of handwriting on the scene image of the display unit of the target terminal device, and the marker information is obtained based on the input unit;

[0116] 2D coordinate point set acquisition module 902 is used to acquire the 2D coordinate point set of the marker information on the scene image;

[0117] The initial background image setting module 903 is used to obtain the minimum bounding rectangle of the 2D coordinate point set and set the image portion of the scene image within the minimum bounding rectangle as the initial background image.

[0118] The mapping module 904 is used to obtain a mapped background image and a mapped coordinate point set based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle. The image content of the mapped background image is the same as the image content of the initial background image, and the relative position of the mapped coordinate point set and the mapped background image is the same as the relative position of the set of 2D coordinate points and the initial background image.

[0119] The real-time scene image acquisition module 905 is used to acquire real-time scene images of the display units of all the terminal devices once every preset time period.

[0120] The homography matrix acquisition module 906 is used to acquire the homography matrix H between the mapped background image and each of the real-time scene images, wherein the homography matrix H makes any pixel coordinate x in the mapped background image and the corresponding pixel coordinate x′ in the real-time scene image satisfy the relationship x′=Hx;

[0121] The real-time 2D coordinate point set acquisition module 907 is used to acquire the real-time 2D coordinate point set corresponding to each terminal device based on the homography matrix H and the mapped coordinate point set.

[0122] The marker information synchronization module 908 is used to render each set of real-time 2D coordinate points into a continuous curve and display it on the real-time scene image of the display unit of each corresponding terminal device, thereby completing the synchronization of the marker information.

[0123] In some embodiments, the 2D coordinate point set acquisition module 902 includes a coordinate point count acquisition unit and a coordinate point set acquisition unit. The coordinate point count acquisition unit acquires a preset number of coordinate points; the coordinate point set acquisition unit acquires points on the marking information at equal or unequal intervals based on the number of coordinate points, and saves them to obtain a 2D coordinate point set of the marking information on the scene image.

[0124] In some embodiments, the mapping module 904 includes a mapping background image acquisition unit and a mapping coordinate point set acquisition unit. The mapping background image acquisition unit is used to map the scene image using a transformation matrix A to obtain a mapping background image; the mapping coordinate point set acquisition unit is used to map the 2D coordinate point set using the transformation matrix A to obtain a mapping coordinate point set. The transformation matrix A is obtained based on the following formula:

[0125]

[0126] Where w is the width of the minimum bounding rectangle, h is the height of the minimum bounding rectangle, θ is the rotation angle of the minimum bounding rectangle, and c x c is the x-value of the center coordinate of the smallest bounding rectangle. y The y-value is the center coordinate of the smallest bounding rectangle.

[0127] It is understood that each component of the tag information synchronization device provided in this application can respectively implement the function of any of the tag information synchronization methods provided in any of the above embodiments, and the specific structure will not be described in detail.

[0128] This application embodiment also provides a remote collaboration system, which includes at least two terminal devices. Each terminal device includes a display unit, an input unit, a sending unit, a receiving unit, and a control and calculation unit. The terminal devices include a receiving terminal device and a sending terminal device.

[0129] In the sending terminal device, please refer to Figure 10 When its control and arithmetic unit receives a confirmation instruction for the tag information issued by the input unit, it executes the following steps:

[0130] S201. Obtain the marking information displayed in the scene image in the display unit;

[0131] S202. Obtain the set of 2D coordinate points of the marked information on the scene image;

[0132] S203. Obtain the minimum bounding rectangle of the 2D coordinate point set, and set the image portion of the scene image within the minimum bounding rectangle as the initial background image;

[0133] S204. Based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle, obtain a mapped background image and a set of mapped coordinate points, wherein the image content of the mapped background image is the same as the image content of the initial background image, and the relative position of the set of mapped coordinate points and the mapped background image is the same as the relative position of the set of 2D coordinate points and the initial background image.

[0134] S205. Bind the mapped background image and the mapped coordinate point set to obtain a reference dataset;

[0135] S206. The reference dataset is sent to all the receiving terminal devices through the sending unit;

[0136] S207. After each first preset time interval, acquire the first real-time scene image of the display unit;

[0137] S208. Obtain the first homography matrix H1 between the mapped background image and the first real-time scene image, wherein the homography matrix H1 such that any pixel coordinate x1 in the mapped background image and the corresponding pixel coordinate x1′ in the first real-time scene image satisfy x1 ′ =H1*x1 relationship;

[0138] S209. Based on the homography matrix H1 and the set of mapped coordinate points, obtain the first real-time 2D coordinate point set;

[0139] S210. Render the first set of real-time 2D coordinate points into a continuous curve and display it on the scene image of the display unit;

[0140] In the receiving terminal device, please refer to Figure 11 When its receiving unit receives the reference dataset, the control operation unit performs the following steps:

[0141] S301. Save the reference dataset;

[0142] S302. After a second preset time interval, acquire a second real-time scene image of the display unit once;

[0143] S303. Obtain the second homography matrix H2 between the mapped background image and the second real-time scene image, wherein the homography matrix H2 such that any pixel coordinate x1 in the mapped background image and the corresponding pixel coordinate x2′ in the first real-time scene image satisfy x2 ′ =H2*x1 relationship;

[0144] S304. Based on the second homography matrix H2 and the set of mapped coordinate points, obtain the second real-time 2D coordinate point set;

[0145] S305. Render the second set of real-time 2D coordinate points into a continuous curve and display it on the scene image of the display unit.

[0146] As described in steps S201-S210 and S301-S305 above, in this embodiment, for the portability of data transmission and storage, the mapped background image and the set of mapped coordinate points are bound together to obtain a reference dataset. This reference dataset is used as the data transmission object for data transmission between the sending terminal device and the receiving terminal device. It should be noted that the working principle of the execution steps of the calculation control unit of the receiving terminal device and the calculation control unit of the sending terminal device in this embodiment is the same as... Figure 9 The working principle of the synchronization device for the marker information shown is similar, and will not be repeated here. Please refer to the relevant descriptions above for details.

[0147] Please refer to Figure 12 , Figure 12 A schematic diagram of a remote collaboration system is shown. Figure 12 In the illustrated embodiment, the remote collaboration system 100 includes a terminal device 10 and a terminal device 20. In this embodiment, when the terminal device 10 is the sender, the terminal device 20 is the receiver, and when the terminal device 10 is the receiver, the terminal device 20 is the sender.

[0148] In some embodiments, please refer again Figure 12 The terminal device also includes a camera, which is used to acquire real-time scene images and transmit them to the display unit for display; the remote collaboration system 100 also includes a server 30, which is used to receive and save reference datasets from the sending terminal device, and to send the reference datasets to the receiving terminal device.

[0149] Please refer to Figure 13 This application also provides a computer device, which includes a processor, memory, network interface, and database connected via a system bus. The processor in this computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores operating devices, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database of the computer device is used to store images, videos, etc. The network interface of the computer device is used for communication with external terminals via a network connection.

[0150] Furthermore, the aforementioned computer device may also be equipped with an input device and a display screen, etc. When the aforementioned computer program is executed by a processor to implement a method for synchronizing marker information, the method includes the following steps: acquiring marker information of a target terminal device, wherein the marker information is displayed in the form of handwriting on a scene image of the display unit of the target terminal device, and the marker information is obtained based on the input unit; acquiring a set of 2D coordinate points of the marker information on the scene image; acquiring the minimum bounding rectangle of the 2D coordinate point set, and setting the image portion of the scene image within the minimum bounding rectangle as an initial background image; based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle, obtaining a mapped background image and a mapped coordinate point set, wherein the image content of the mapped background image is the same as the image content of the initial background image, and the mapped coordinate point set... The relative position of the mapped background image is the same as the relative position of the 2D coordinate point set and the initial background image; after a preset time interval, the real-time scene image of the display units of all the terminal devices is acquired once; the homography matrix H of the initial background image and each of the real-time scene images is acquired, wherein the homography matrix H makes any pixel coordinate x in the mapped background image and the corresponding pixel coordinate x′ in the real-time scene image satisfy the relationship x′=Hx; based on the homography matrix H and the mapped coordinate point set, the real-time 2D coordinate point set corresponding to each terminal device is acquired; each of the real-time 2D coordinate point sets is rendered as a continuous curve and displayed on the real-time scene image of the corresponding display unit of each terminal device, thus completing the synchronization of the marking information.

[0151] This application embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements a method for synchronizing marker information. The method includes the following steps: acquiring marker information of a target terminal device, wherein the marker information is displayed in the form of handwriting on a scene image of the display unit of the target terminal device, and the marker information is obtained based on the input unit; acquiring a set of 2D coordinate points of the marker information on the scene image; acquiring the minimum bounding rectangle of the 2D coordinate point set, and setting the image portion of the scene image within the minimum bounding rectangle as an initial background image; based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle, obtaining a mapped background image and a mapped coordinate point set, wherein the image content of the mapped background image is the same as that of the initial background image. Similar to the content, the relative positions of the mapped coordinate point set and the mapped background image are the same as the relative positions of the 2D coordinate point set and the initial background image; after a preset time interval, the real-time scene image of the display units of all the terminal devices is acquired once; the homography matrix H of the initial background image and each of the real-time scene images is acquired, wherein the homography matrix H makes any pixel coordinate x in the mapped background image and the corresponding pixel coordinate x′ in the real-time scene image satisfy the relationship x′=Hx; based on the homography matrix H and the mapped coordinate point set, the real-time 2D coordinate point set corresponding to each terminal device is acquired; each of the real-time 2D coordinate point sets is rendered as a continuous curve and displayed on the real-time scene image of the corresponding display unit of each terminal device, thus completing the synchronization of the marking information.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media provided in this application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-rate SDRAM (SSRSDRAM), expanded SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0153] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0154] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for synchronizing marker information, characterized in that, For a remote collaboration system, the remote collaboration system including at least two terminal devices, each terminal device including a display unit and an input unit, the method includes: Acquire marking information of the target terminal device, wherein the marking information is displayed in the form of handwriting on the scene image of the display unit of the target terminal device, and the marking information is obtained based on the input unit; Obtain the set of 2D coordinate points of the marked information on the scene image; Obtain the minimum bounding rectangle of the 2D coordinate point set, and set the image portion of the scene image within the minimum bounding rectangle as the initial background image; Based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle, a mapped background image and a set of mapped coordinate points are obtained. The image content of the mapped background image is the same as the image content of the initial background image, and the relative position of the set of mapped coordinate points and the mapped background image is the same as the relative position of the set of 2D coordinate points and the initial background image. Every preset time interval, a real-time scene image of the display units of all the terminal devices is acquired once; Obtain the homography matrix H of the initial background image and each of the real-time scene images, wherein the homography matrix H makes any pixel coordinate x in the mapped background image and the corresponding pixel coordinate x′ in the real-time scene image satisfy the relationship x′=Hx; Based on the homography matrix H and the set of mapped coordinate points, obtain the set of real-time 2D coordinate points corresponding to each terminal device; Each set of real-time 2D coordinate points is rendered as a continuous curve and displayed on the real-time scene image of the display unit of each corresponding terminal device, thus completing the synchronization of the marking information.

2. The method for synchronizing tag information according to claim 1, characterized in that, The step of obtaining the set of 2D coordinate points of the marked information on the scene image includes: Get the preset number of coordinate points; Based on the number of coordinate points, points are taken at equal or unequal intervals on the marking information and saved to obtain a set of 2D coordinate points of the marking information on the scene image.

3. The method for synchronizing marker information according to claim 1, characterized in that, The step of obtaining the mapped background image and the mapped coordinate point set based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle includes: The scene image is mapped using transformation matrix A to obtain the mapped background image; The transformation matrix A is used to map the 2D coordinate point set to obtain the mapped coordinate point set; The transformation matrix A is obtained based on the following formula: Where w is the width of the minimum bounding rectangle, h is the height of the minimum bounding rectangle, θ is the rotation angle of the minimum bounding rectangle, and c x Let x be the x-value of the center coordinate of the minimum bounding rectangle, and c be the x-value of the center coordinate of the minimum bounding rectangle. y Let y be the center coordinate of the minimum bounding rectangle.

4. A device for synchronizing marker information, characterized in that, For a remote collaboration system, the remote collaboration system includes at least two terminal devices, each terminal device including a display unit and an input unit, the device comprising: Acquire marking information of the target terminal device, wherein the marking information is displayed in the form of handwriting on the scene image of the display unit of the target terminal device, and the marking information is obtained based on the input unit; Obtain the set of 2D coordinate points of the marked information on the scene image; Obtain the minimum bounding rectangle of the 2D coordinate point set, and set the image portion of the scene image within the minimum bounding rectangle as the initial background image; This is used to obtain a mapped background image and a mapped coordinate point set based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle, wherein the image content of the mapped background image is the same as the image content of the initial background image, and the relative position of the mapped coordinate point set and the mapped background image is the same as the relative position of the set of 2D coordinate points and the initial background image. Every preset time interval, a real-time scene image of the display units of all the terminal devices is acquired once; Used to obtain the homography matrix H between the mapped background image and each of the real-time scene images, wherein the homography matrix H makes any pixel coordinate x in the mapped background image and the corresponding pixel coordinate x′ in the real-time scene image satisfy the relationship x′=Hx; Based on the homography matrix H and the set of mapped coordinate points, obtain the set of real-time 2D coordinate points corresponding to each terminal device; Each set of 2D coordinate points is rendered as a continuous curve and displayed on the scene image of the display unit of each terminal device, thus completing the synchronization of the marking information.

5. A remote collaboration system, characterized in that, include: At least two terminal devices and a controller, the terminal devices including a display unit and an input unit, the controller being used to execute the method for synchronizing tag information according to any one of claims 1-3.

6. A remote collaboration system, characterized in that, include: At least two terminal devices, each terminal device including a display unit, an input unit, a sending unit, a receiving unit, and a control and calculation unit, wherein the at least two terminal devices include a receiving terminal device and a sending terminal device; In the sending terminal device, when the control calculation unit receives the tag information confirmation command issued by the input unit, it performs the following steps: Obtain the marking information displayed in the scene image in the display unit; Obtain the set of 2D coordinate points of the marked information on the scene image; Obtain the minimum bounding rectangle of the 2D coordinate point set, and set the image portion of the scene image within the minimum bounding rectangle as the initial background image; Based on the scene image, the set of 2D coordinate points, and the minimum bounding rectangle, a mapped background image and a set of mapped coordinate points are obtained. The image content of the mapped background image is the same as the image content of the initial background image, and the relative position of the set of mapped coordinate points and the mapped background image is the same as the relative position of the set of 2D coordinate points and the initial background image. The mapped background image and the mapped coordinate point set are bound together to obtain a reference dataset; The reference dataset is sent to all the receiving terminal devices through the sending unit; After a first preset time interval, the first real-time scene image of the display unit is acquired once; Obtain the first homography matrix H1 between the mapped background image and the first real-time scene image, wherein the homography matrix H1 such that any pixel coordinate x1 in the mapped background image and the corresponding pixel coordinate x1′ in the first real-time scene image satisfy x1 ′ =H1*x1 relationship; Based on the homography matrix H1 and the set of mapped coordinate points, a first real-time 2D coordinate point set is obtained; The first set of real-time 2D coordinate points is rendered as a continuous curve and displayed on the scene image of the display unit; In the receiving terminal device, when the receiving unit receives the reference dataset, the control calculation unit performs the following steps: Save the reference dataset; Every second preset time interval, the second real-time scene image of the display unit is acquired once; Obtain the second homography matrix H2 between the mapped background image and the second real-time scene image, wherein the homography matrix H2 such that any pixel coordinate x1 in the mapped background image and the corresponding pixel coordinate x2′ in the first real-time scene image satisfy x2 ′ =H2*x1 relationship; Based on the second homography matrix H2 and the set of mapped coordinate points, obtain the second 2D real-time coordinate point set; The set of 2D real-time coordinate points is rendered as a continuous curve and displayed on the scene image of the display unit.

7. The remote collaboration system according to claim 6, characterized in that, The terminal device also includes a camera, which is used to acquire real-time scene images and transmit them to the display unit for display.

8. The remote collaboration system according to claim 6, characterized in that, The remote collaboration system also includes a server; The server is used to receive and store reference datasets from the sending terminal device, and to send the reference datasets to the receiving terminal device.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for synchronizing tag information according to any one of claims 1-3.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for synchronizing tag information according to any one of claims 1-3.

Images