Remote annotation methods and devices

Abstract

The present application relates to a remote annotation method and device, which is applied to a user end and includes: obtaining local annotation information; sending the local annotation information to a remote server; obtaining remote annotation information from the remote server; generating annotation content on a canvas according to the local annotation information and the remote annotation information; and displaying the annotation content. According to the remote annotation solution provided by the present application, annotations from multiple remote terminals can be displayed in real time to each other, realizing the remote drawing and clearing functions of handwriting from multiple terminals, making the annotation process more interactive and improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of intelligent interaction technology, and in particular to a remote annotation method and device. Background Art

[0002] With the advancement of science and technology and the development of network technology, the remote interaction methods in various fields are constantly changing. It is particularly important to note that the methods and approaches of education are also undergoing rapid transformation, and the efficiency of educational communication is also rapidly improving. Among them, remote classroom education makes educational communication faster and more convenient, and multi-terminal interactive annotations make the interaction in remote education more specific and vivid, and the communication faster and more effective.

[0003] However, currently, annotations from multiple users are still mainly performed locally. At the same time, there are some problems with the handwriting drawing and display of annotations from other users. In order to improve the user experience, remote annotations between multiple users need to be improved. Summary of the invention

[0004] Currently, multi-terminal annotation is mainly performed locally, which results in the inability to share annotation information between multiple user terminals that make annotations, or only the annotations of the main speaker among the user terminals can be shared while the annotations of other listening terminals cannot be displayed; in addition, there are also problems such as the inability to distinguish users when drawing handwriting and the inability to distinguish users when clearing handwriting.

[0005] According to a first aspect of the present application, a remote annotation method is provided, which is applied to a user terminal, and the method comprises:

[0006] Get local annotation information;

[0007] Sending the local annotation information to a remote server;

[0008] Acquiring remote annotation information from the remote server; and

[0009] Generating annotation content on the canvas according to the local annotation information and the remote annotation information;

[0010] The annotation content is displayed.

[0011] According to a second aspect of the present application, a remote annotation device is provided, comprising:

[0012] A first acquisition unit, used to acquire local annotation information;

[0013] A sending unit, used for sending the local annotation information to a remote server;

[0014] a second acquiring unit, configured to acquire remote annotation information from the remote server; and

[0015] An annotation generating unit, used for generating annotation content on the canvas according to the local annotation information and the remote annotation information;

[0016] A display unit is used to display the annotation content.

[0017] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0018] Processor; and

[0019] A memory storing a computer program, which, when executed by the processor, enables the processor to execute the method described in the first aspect.

[0020] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having computer-readable instructions stored thereon, and when the instructions are executed by a processor, the processor is caused to execute the method as described in the first aspect.

[0021] According to the remote annotation solution provided by the present application, the annotations in the remote multiple terminals (including the main speaker terminal and the listening terminal) can be displayed to each other in real time, realizing the remote drawing and clearing functions of the multi-terminal handwritings, making the multi-terminal annotation process more interactive and improving the user experience; in addition, for different users or different handwritings, different parameters are defined for the drawn handwritings to distinguish the users or annotations; in addition, different users and different handwritings have corresponding user identifiers and handwriting identifiers, so that in the process of clearing the handwritings, the corresponding handwritings can be cleared according to the user identifier and the handwriting identifier, and different clearing permissions can be set for each terminal of the remote multiple terminals. For example, according to the user identifier, the main speaker terminal can clear its own handwriting, other handwritings and all handwritings, and the listening terminal can only clear its own handwriting. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without exceeding the scope of protection required by the present application.

[0023] Figure 1 It is a flowchart of a remote annotation method according to an embodiment of the present application.

[0024] Figure 2 It is a flowchart of obtaining the current user's annotation information in the drawing mode in the remote annotation method according to an embodiment of the present application.

[0025] Figure 3 It is a flowchart of obtaining the current user's annotation information in the clearing mode in the remote annotation method according to an embodiment of the present application.

[0026] Figure 4 It is a flowchart of requesting annotation information according to a handwriting update message in a remote annotation method according to an embodiment of the present application.

[0027] Figure 5 It is a flowchart of obtaining remote annotation information of a remote server in a remote annotation method according to an embodiment of the present application.

[0028] Figure 6 It is a flowchart of generating annotation content in the remote annotation method according to an embodiment of the present application.

[0029] Figure 7 is a schematic diagram of a remote annotation device according to an embodiment of the present application.

[0030] Figure 8 It is a structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0032] The present application relates to a remote annotation solution, which can be applied to the scenario of a remote multi-terminal classroom, and can also be applied to other scenarios of multiple user terminals interacting. In the specification, the remote multi-terminal classroom is used as an example to describe the solution. In a remote multi-terminal classroom, the user terminals (including the lecture terminal and the listening terminal) are connected through a remote server, and annotations are made during interaction. The lecture terminal and the listening terminal send local annotation information to the remote server, and receive annotation information from other terminals from the remote server, thereby realizing the remote drawing and clearing function of multi-terminal handwriting. In the present application, the user terminal connected through the remote server can be any electronic device with a display function, such as a personal computer, a mobile phone, a tablet computer, etc., without any restriction.

[0033] The main process of the present application scheme is: any user terminal among multiple user terminals can turn on the annotation function, display the annotation screen and setting menu, and the setting menu includes drawing mode (handwriting drawing mode and handwriting clearing mode), brush setting and exit annotation and other functions. Select the drawing mode, create a handwriting object for each annotation handwriting, record the drawing mode, the user ID of the annotation handwriting, the ID of the annotation handwriting, the brush setting parameters, the handwriting path when the drawing mode is the drawing mode or the clearing area when the drawing mode is the clearing mode, the node type of the point contained in the handwriting path, the node type of the point contained in the clearing mode and other information in the handwriting object. On the one hand, the local user terminal obtains the local handwriting object corresponding to the local annotation handwriting, generates the annotation content on the canvas according to the local handwriting object, and sends the local handwriting object data to the remote server. On the other hand, it receives the remote handwriting object data from other user terminals from the remote server, and generates the annotation content on the canvas according to the remote handwriting object data.

[0034] As described above, the solution of the present application generally includes three aspects: local annotation acquisition, remote annotation acquisition, and generation of annotation content. Figure 1 (or Figure 7 ) generally discloses the process of remote annotation scheme, Figures 2 to 6 The processes of obtaining the current user's annotation information in the drawing mode, obtaining the current user's annotation information in the clearing mode, requesting annotation information according to the handwriting update message, obtaining the remote annotation information of the remote server, and generating the annotation content are disclosed respectively.

[0035] According to one aspect of the present application, a remote annotation method is provided. Figure 1 is a flow chart of a remote annotation method according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0036] Step S101, obtaining local annotation information.

[0037] The drawing modes of annotation traces include drawing mode and clearing mode. Figure 1 and Figure 2 The flowcharts of obtaining the current user annotation information in the drawing mode and in the clearing mode are described respectively.

[0038] like Figure 1As shown, the local end displays the background canvas, annotation screen and setting menu. The setting menu includes drawing mode (handwriting drawing mode and handwriting clearing mode), brush settings and exit annotation and other functions. In response to the touch event, the absolute coordinates are obtained for calculating the touch coordinates. When the drawing mode is selected in the setting menu, the touch event is processed. The touch event includes three event types: press event, move event and lift event. Obtaining local annotation information includes obtaining the handwriting object of the local annotation handwriting. The handwriting object includes the drawing mode, handwriting path and clearing area of ​​the annotation handwriting.

[0039] In the drawing mode, step S101 includes the following steps:

[0040] Sub-step S1011, in response to a press event, recording the coordinates of the point corresponding to the press event in the handwriting path.

[0041] like Figure 2 As shown, in response to a press event, a handwriting object of the annotation handwriting is initialized, a new handwriting path is created, the coordinates of the point corresponding to the press event are recorded in the handwriting path, and the handwriting path is added to the local cache.

[0042] Sub-step S1012, in response to the movement event, recording the coordinates of the point corresponding to the movement event in the handwriting path.

[0043] like Figure 2 As shown, in response to a movement event, the coordinates of the point corresponding to the movement event are recorded in the handwriting path, and the handwriting path is added to the local cache.

[0044] Sub-step S1013, in response to the lift-up event, recording the coordinates of the point corresponding to the lift-up event in the handwriting path.

[0045] like Figure 2 As shown, in response to a lift event, the coordinates of the point corresponding to the lift event are recorded in the handwriting path, and the handwriting path is added to the local cache.

[0046] In one embodiment, when a lift event is detected, the handwriting corresponding to the entire handwriting path is drawn on the canvas. Figure 2 As shown, in the process of recording the coordinates of the point corresponding to the pressing event, the coordinates of the point corresponding to the moving event and the coordinates of the point corresponding to the lifting event in the handwriting path, the handwriting corresponding to the handwriting path is drawn on the canvas. In this way, it is possible to draw and display at the same time in the drawing mode.

[0047] like Figure 3 As shown, when the clear mode is selected in the setting menu, touch events are processed, and touch events also include three event types: press events, move events, and lift events.

[0048] In the clearing mode, step S101 includes the following steps:

[0049] Sub-step S1014, in response to the pressing event, recording the coordinates of the point corresponding to the pressing event in the clearing area.

[0050] like Figure 3 As shown, in response to a press event, the handwriting object of the annotation handwriting is initialized, a new clearing area is created, the coordinates of the point corresponding to the press event are recorded in the clearing area, and the clearing area is added to the local cache.

[0051] Sub-step S1015, in response to the motion event, recording the coordinates of the point corresponding to the motion event in the clearing area.

[0052] like Figure 3 As shown, in response to a movement event, the coordinates of the point corresponding to the movement event are recorded in the clearing area, and the clearing area is added to the local cache.

[0053] Sub-step S1016, in response to the lift event, recording the coordinates of the point corresponding to the lift event in the clearing area.

[0054] like Figure 3 As shown, in response to a lift event, the coordinates of the point corresponding to the lift event are recorded in the clearing area, and the clearing area is added to the local cache.

[0055] In one embodiment, when a lifting event is detected, the handwriting that intersects with the clearing area is removed from the canvas. In this way, when a lifting event is triggered, the handwriting selected by the clearing area can be deleted.

[0056] In one embodiment, the handwriting object of the annotation handwriting also includes node type information, and the node type information is used to classify the points corresponding to the pressing event, the moving event, and the lifting event of the annotation handwriting, and determine these points as pressing points, moving points, and lifting points, respectively, so that after the local handwriting object is sent to the remote end, the remote end can accurately draw the handwriting corresponding to the handwriting object according to the node type information. In this way, step S101 also includes:

[0057] Sub-step S1017, determining the node types of the point corresponding to the press event, the point corresponding to the move event, and the point corresponding to the lift event as a press point, a move point, and a lift point, respectively, and recording them in the node type information.

[0058] In one embodiment, the handwriting object of the annotation handwriting also includes the user identification of the annotation handwriting, the handwriting identification of the annotation handwriting, and the pen setting parameters of the annotation handwriting. The user identification is used to identify which user made the annotation handwriting, the handwriting identification is used to distinguish different annotation handwritings, and the pen setting parameters are used to set the properties of the brush, such as the type of brush (for example, the shape of the pen, the shape of the eraser), color, thickness, etc.

[0059] Step S102: Send the local annotation information to the remote server.

[0060] Send local annotation information to the remote server to facilitate other terminals to obtain local annotation information from the server and share local annotation information with other terminals.

[0061] In one embodiment, in the drawing mode, when a lift event is detected, the entire handwriting object information is sent to a remote server, and other terminals obtain local handwriting path information through the remote server.

[0062] In another embodiment, in the drawing mode, in the process of recording the coordinates of the point corresponding to the pressing event, the coordinates of the point corresponding to the moving event and the coordinates of the point corresponding to the lifting event in the handwriting path of the handwriting object, the handwriting path is sent to the remote server, and other terminals obtain the local handwriting path information through the remote server. In this way, the drawing can be done locally while the remote terminal can display the drawing process in real time.

[0063] In one embodiment, in the clearing mode, when a lift event is detected, the entire handwriting object information is sent to the remote server, and the other end obtains the local handwriting path information through the remote server. In this way, the other end can delete the handwriting selected in the clearing area.

[0064] Step S103, obtaining remote annotation information from a remote server.

[0065] In one embodiment, the speaker terminal or the listener terminal updates the handwriting according to the handwriting update message sent from the remote server.

[0066] When the remote server receives the annotation handwriting message from either the main speaker or the listening end, it will send an update handwriting message to the other end or the selected end. When the main speaker and the listening end receive the update handwriting message, they will actively obtain the annotation information from the remote server. Since the obtained annotation information usually includes the information of multiple annotation handwritings of multiple users, the obtained information is also called the annotation information set.

[0067] Thus, step S103 includes: step S1031, in response to the handwriting update message from the remote server, obtaining the annotation information set from the remote server.

[0068] Figure 4 1 is a flow chart of requesting annotation information according to a handwriting update message in a remote annotation method according to an embodiment of the present application. Figure 4 In the present invention, the MQTT (Message Queuing Telemetry Transport) mechanism is used as an example to describe the way in which the speaker and the listener request annotation information according to the handwriting update message. It should be noted that other mechanisms can also be used to realize the request for annotation information according to the handwriting update message.

[0069] like Figure 4 As shown, the method includes: 1) when the annotation function is started, the main speaker or the listening end completes the MQTT link as a client and registers for handwriting update message listening; 2) when the main speaker or the listening end makes an annotation, the annotation information will be uploaded to the remote server, and then the remote server will send an MQTT message to the online client in the managed MQTT client list to notify the client to update all handwritings; 3) after receiving the MQTT message for updating the handwriting, the MQTT client will update the remote handwriting, update the local handwriting cache after receiving the annotation information set, and redraw it on the canvas.

[0070] In another embodiment, the speaker terminal or the listener terminal may actively obtain the remote annotation handwriting when no handwriting update message is received, for example, obtaining the remote annotation handwriting during the local initialization stage.

[0071] After receiving the handwriting update message, how does the speaker or listener obtain the handwriting information from the remote server? Figure 5 A detailed description is given in . Figure 5 This is a flow chart of obtaining remote annotation information of a remote server in a remote annotation method according to an embodiment of the present application. Figure 5 In the process of sending the handwriting update message, the remote server sends more than one handwriting update message to the main speaker or the listening end. In the process of sending the handwriting update message, the handwriting update message can be distinguished by, for example, a sequence number and the order of each handwriting update message can be indicated. Generally speaking, the handwriting update message sent later is more updated, and the sequence number of the more updated handwriting update message can be marked larger. The main speaker or the listening end will store the current handwriting update message and its sequence number (called "current sequence number"). When the main speaker or the listening end obtains the handwriting information from the remote server after receiving the handwriting update message, the local (main speaker or the listening end) state needs to be considered. The local state includes an idle state and an acquisition state, wherein the acquisition state indicates that the main speaker or the listening end is obtaining the handwriting information set corresponding to the current sequence number from the remote server, and the idle state indicates that the main speaker or the listening end is not obtaining the handwriting information set corresponding to the current sequence number from the remote server, and is in a state of waiting for update.

[0072] like Figure 5 As shown, the main speaker or the listening end receives the MQTT message, obtains the handwriting sequence number and sets it as the MQTT message sequence number, and then checks whether the main speaker or the listening end is in an idle state.

[0073] When it is not in the idle state, that is, in the acquiring state, no matter whether the MQTT message sequence number is greater than the current sequence number (that is, no matter whether the MQTT message is a message that is newer than the update message indicated by the current sequence number), the annotation information set corresponding to the current handwriting update message is obtained from the remote server; after the annotation information set data is acquired, the state is set to the idle state, and the data with the current sequence number as the starting point and the MQTT message sequence number as the end point is acquired to determine whether the starting point is greater than or equal to the end point. In this case, if the starting point is less than the end point, the state is set to the acquiring state, and the handwriting information set corresponding to the MQTT message (sequence number) is acquired from the remote server.

[0074] When in idle state, obtain data with the current sequence number as the starting point and the MQTT message sequence number as the end point, and determine whether the starting point is greater than or equal to the end point. When the starting point is less than the end point, set the state to the acquisition state, and obtain the handwriting information set corresponding to the MQTT message sequence number from the remote server; when the starting point is greater than or equal to the end point, continue to be in idle state, and end the acquisition process corresponding to the MQTT message.

[0075] exist Figure 5 In the process, whether obtaining the annotation information set corresponding to the current handwriting update message or the handwriting information set corresponding to the MQTT message from the remote server, when the data is successfully obtained, the obtained data (handwriting information set) is parsed, the drawn node list information is obtained, the handwriting point information in the node list information is added to the drawn handwriting, the local cache is updated, and the main speaker or the listening end updates the displayed handwriting according to the updated local cache.

[0076] exist Figure 5 In the process, whether obtaining the annotation information set corresponding to the current handwriting update message or obtaining the handwriting information set corresponding to the MQTT message from the remote server, when the data acquisition is unsuccessful, the acquisition process is terminated.

[0077] Thus, according to Figure 5 In the process shown, step S1031 includes the following sub-steps:

[0078] Sub-step S10311, in response to being in the acquisition state, acquiring the annotation information set corresponding to the current handwriting update message from the remote server; and

[0079] Sub-step S10312, in response to being in an idle state, when the updated handwriting message from the remote server is an updated handwriting update message, obtain the annotation information set corresponding to the updated handwriting message from the remote server from the remote server; otherwise, obtain the annotation information set corresponding to the current handwriting update message from the remote server.

[0080] After receiving the annotation information set from the remote server, the speaker and the listener need to process the handwriting information set. The processing includes:

[0081] 1. Whether the annotation information set obtained from the remote server belongs to the division area where the current speaker or listener is located.

[0082] For example, in the application scenario of remote multi-terminal classrooms, the remote server can manage more than one remote classroom, and there may be a situation where the annotation information set of one remote classroom is sent to another remote classroom. Therefore, after receiving the annotation information set, the lecturer or the lecturer needs to determine whether the annotation information set belongs to the classroom to which the lecturer or the lecturer belongs. If it belongs to the classroom, the annotation information set is parsed; if it does not belong, the annotation information set is discarded.

[0083] 2. Determine whether the obtained annotation information set is empty.

[0084] For example, a specific algorithm can be used to determine whether the annotation information set is empty. If it is empty, no further parsing is required; if it is not empty, the internal data can be parsed.

[0085] 3. Obtain the display ratio data of the annotation information set and the display ratio data of the current speaker and listener, and display the handwriting information of the annotation information set after conversion.

[0086] Before displaying the handwriting information of the annotation information set, in order to correctly display the annotation handwriting, it is necessary to ensure that the display ratio of the received annotation information set is consistent with the display ratio of the current speaker end and the listener end.

[0087] Step S104: Generate annotation content on the canvas according to the local annotation information and the remote annotation information.

[0088] When new local annotation handwriting appears or new remote annotation handwriting is received, the speaker or listener needs to update the handwriting on the canvas.

[0089] When the annotation content is generated on the canvas according to the local annotation information, as described above, in one embodiment, when a lift event is detected, the handwriting corresponding to the entire handwriting path is drawn on the canvas. Figure 2As shown, in the process of recording the coordinates of the point corresponding to the pressing event, the coordinates of the point corresponding to the moving event and the coordinates of the point corresponding to the lifting event in the handwriting path, the handwriting corresponding to the handwriting path is drawn on the canvas. In this way, it is possible to draw and display at the same time in the drawing mode.

[0090] Thus, step S104 includes: sub-step S1041, in the process of recording the coordinates of the point corresponding to the pressing event, the coordinates of the point corresponding to the moving event and the coordinates of the point corresponding to the lifting event in the handwriting path, drawing the handwriting corresponding to the handwriting path on the canvas.

[0091] When the annotation content is generated on the canvas according to the local annotation information, as described above, in one embodiment, when a lift event is detected, the handwriting that intersects with the clearing area is cleared on the canvas. In this way, when the lift event is triggered, the handwriting selected in the clearing area can be deleted.

[0092] Therefore, step S104 includes: sub-step S1042, in response to the lift event, clearing the handwriting that intersects with the clearing area on the canvas.

[0093] In the process of generating annotation content on the canvas according to the remote annotation information, it is necessary to generate annotation content on the canvas according to the handwriting object in the remote annotation information. In this process, it is necessary to use the node type information in the handwriting object to accurately draw the handwriting in the remote annotation information.

[0094] When new local annotation handwriting appears or new remote annotation handwriting is received, the speaker or listener needs to update the handwriting, which is mainly completed by traversing the handwriting list of the current user and other users in the local buffer. Figure 6 It is a flowchart of generating annotation content in the remote annotation method according to an embodiment of the present application.

[0095] like Figure 6 As shown, the process of generating annotation content includes: 1) determining whether there is a temporary bitmap. If there is no temporary bitmap, creating a new temporary bitmap and canvas; 2) clearing the canvas content. In one embodiment, the color of the canvas can be set to transparent; 3) traversing the handwriting objects corresponding to the local annotation information of the current user in the local cache, and generating annotation content on the canvas; 4) traversing the handwriting objects corresponding to the remote annotation information of other users in the local cache, and generating annotation content on the canvas; 5) drawing a temporary bitmap in the canvas, and displaying the annotation content generated by the canvas on the screen.

[0096] Therefore, the remote annotation method provided by this application also includes:

[0097] Step S105: display the annotation content.

[0098] According to drawing a temporary bitmap in the canvas, the annotation content generated by the canvas is displayed on the screen of the user end.

[0099] According to the remote annotation method provided by the present application, annotations in multiple remote terminals can be displayed to each other in real time, realizing the remote drawing and clearing functions of handwritings on multiple terminals, making the multi-terminal annotation process more interactive and improving the user experience.

[0100] According to another aspect of the present application, a remote annotation device is provided. Figure 7 Schematic diagram of a remote annotation device according to an embodiment of the present application. Figure 7 As shown, the device includes the following units:

[0101] The first acquisition unit 701 is used to acquire local annotation information.

[0102] The drawing modes of annotation traces include drawing mode and clearing mode. Figure 1 and Figure 2 The flowcharts of obtaining the current user annotation information in the drawing mode and in the clearing mode are described respectively.

[0103] like Figure 1 As shown, the local end displays the background canvas, annotation screen and setting menu. The setting menu includes drawing mode (handwriting drawing mode and handwriting clearing mode), brush settings and exit annotation and other functions. In response to the touch event, the absolute coordinates are obtained for calculating the touch coordinates. When the drawing mode is selected in the setting menu, the touch event is processed. The touch event includes three event types: press event, move event and lift event. Obtaining local annotation information includes obtaining the handwriting object of the local annotation handwriting. The handwriting object includes the drawing mode, handwriting path and clearing area of ​​the annotation handwriting.

[0104] In the drawing mode, the first acquisition unit 701 includes the following subunits:

[0105] The first recording subunit 7011 is used to record the coordinates of the point corresponding to the pressing event in the handwriting path in response to the pressing event.

[0106] like Figure 2 As shown, in response to a press event, a handwriting object of the annotation handwriting is initialized, a new handwriting path is created, the coordinates of the point corresponding to the press event are recorded in the handwriting path, and the handwriting path is added to the local cache.

[0107] The second recording subunit 7012 is used to record the coordinates of the point corresponding to the movement event in the handwriting path in response to the movement event.

[0108] like Figure 2As shown, in response to a movement event, the coordinates of the point corresponding to the movement event are recorded in the handwriting path, and the handwriting path is added to the local cache.

[0109] The third recording subunit 7013 is used to record the coordinates of the point corresponding to the lift-up event in the handwriting path in response to the lift-up event.

[0110] like Figure 2 As shown, in response to a lift event, the coordinates of the point corresponding to the lift event are recorded in the handwriting path, and the handwriting path is added to the local cache.

[0111] In one embodiment, when a lift event is detected, the handwriting corresponding to the entire handwriting path is drawn on the canvas. Figure 2 As shown, in the process of recording the coordinates of the point corresponding to the pressing event, the coordinates of the point corresponding to the moving event and the coordinates of the point corresponding to the lifting event in the handwriting path, handwriting corresponding to the annotation content path is generated on the canvas. In this way, it is possible to draw and display at the same time in the drawing mode.

[0112] like Figure 3 As shown, when the clear mode is selected in the setting menu, touch events are processed, and touch events also include three event types: press events, move events, and lift events.

[0113] In the clearing mode, the first acquisition unit 701 includes the following subunits:

[0114] The fourth recording subunit 7014 is used to record the coordinates of the point corresponding to the pressing event in the clearing area in response to the pressing event.

[0115] like Figure 3 As shown, in response to a press event, the handwriting object of the annotation handwriting is initialized, a new clearing area is created, the coordinates of the point corresponding to the press event are recorded in the clearing area, and the clearing area is added to the local cache.

[0116] The fifth recording subunit 7015 is used to record the coordinates of the point corresponding to the movement event in the clearing area in response to the movement event.

[0117] like Figure 3 As shown, in response to a movement event, the coordinates of the point corresponding to the movement event are recorded in the clearing area, and the clearing area is added to the local cache.

[0118] The sixth recording subunit 7016 is used to record the coordinates of the point corresponding to the lift event in the clearing area in response to the lift event.

[0119] like Figure 3As shown, in response to a lift event, the coordinates of the point corresponding to the lift event are recorded in the clearing area, and the clearing area is added to the local cache.

[0120] In one embodiment, when a lifting event is detected, the handwriting that intersects with the clearing area is removed from the canvas. In this way, when a lifting event is triggered, the handwriting selected by the clearing area can be deleted.

[0121] In one embodiment, the handwriting object of the annotation handwriting also includes node type information, and the node type information is used to classify the points corresponding to the pressing event, the moving event, and the lifting event of the annotation handwriting, and determine these points as pressing points, moving points, and lifting points, respectively, so that after the local handwriting object is sent to the remote end, the remote end can accurately draw the handwriting corresponding to the handwriting object according to the node type information. In this way, the first acquisition unit 701 also includes:

[0122] The seventh recording subunit 7017 is used to determine the node types of the point corresponding to the pressing event, the point corresponding to the moving event and the point corresponding to the lifting event as a pressing point, a moving point and a lifting point respectively, and record them in the node type information.

[0123] In one embodiment, the handwriting object of the annotation handwriting also includes the user identification of the annotation handwriting, the handwriting identification of the annotation handwriting, and the pen setting parameters of the annotation handwriting. The user identification is used to identify which user made the annotation handwriting, the handwriting identification is used to distinguish different annotation handwritings, and the pen setting parameters are used to set the properties of the brush, such as the type of brush (for example, the shape of the pen, the shape of the eraser), color, thickness, etc.

[0124] The sending unit 702 is used to send the local annotation information to the remote server.

[0125] Send local annotation information to the remote server to facilitate other terminals to obtain local annotation information from the server and share local annotation information with other terminals.

[0126] In one embodiment, in the drawing mode, when a lift event is detected, the entire handwriting object information is sent to a remote server, and other terminals obtain local handwriting path information through the remote server.

[0127] In another embodiment, in the drawing mode, in the process of recording the coordinates of the point corresponding to the pressing event, the coordinates of the point corresponding to the moving event and the coordinates of the point corresponding to the lifting event in the handwriting path of the handwriting object, the handwriting path is sent to the remote server, and other terminals obtain the local handwriting path information through the remote server. In this way, the drawing can be done locally while the remote terminal can display the drawing process in real time.

[0128] In one embodiment, in the clearing mode, when a lift event is detected, the entire handwriting object information is sent to the remote server, and the other end obtains the local handwriting path information through the remote server. In this way, the other end can delete the handwriting selected in the clearing area.

[0129] The second acquisition unit 703 is used to acquire remote annotation information from a remote server.

[0130] In one embodiment, the speaker terminal or the listener terminal updates the handwriting according to the handwriting update message sent from the remote server.

[0131] When the remote server receives the annotation handwriting message from either the main speaker or the listening end, it will send an update handwriting message to the other end or the selected end. When the main speaker and the listening end receive the update handwriting message, they will actively obtain the annotation information from the remote server. Since the obtained annotation information usually includes the information of multiple annotation handwritings of multiple users, the obtained information is also called the annotation information set.

[0132] Thus, the second acquisition unit 703 includes: an acquisition subunit 7031, which is used to acquire a set of annotation information from the remote server in response to an update handwriting message from the remote server.

[0133] Figure 4 1 is a flow chart of requesting annotation information according to a handwriting update message in a remote annotation method according to an embodiment of the present application. Figure 4 In the present invention, the MQTT (Message Queuing Telemetry Transport) mechanism is used as an example to describe the way in which the speaker and the listener request annotation information according to the handwriting update message. It should be noted that other mechanisms can also be used to realize the request for annotation information according to the handwriting update message.

[0134] like Figure 4 As shown, the method includes: 1) when the annotation function is started, the main speaker or the listening end completes the MQTT link as a client and registers for handwriting update message listening; 2) when the main speaker or the listening end makes an annotation, the annotation information will be uploaded to the remote server, and then the remote server will send an MQTT message to the online client in the managed MQTT client list to notify the client to update all handwritings; 3) after receiving the MQTT message for updating the handwriting, the MQTT client will update the remote handwriting, update the local handwriting cache after receiving the annotation information set, and redraw it on the canvas.

[0135] In another embodiment, the speaker terminal or the listener terminal may actively obtain the remote annotation handwriting when no handwriting update message is received, for example, obtaining the remote annotation handwriting during the local initialization stage.

[0136] After receiving the handwriting update message, how does the speaker or listener obtain the handwriting information from the remote server? Figure 5 A detailed description is given in . Figure 5 This is a flow chart of obtaining remote annotation information of a remote server in a remote annotation method according to an embodiment of the present application. Figure 5 In the process of sending the handwriting update message, the remote server sends more than one handwriting update message to the main speaker or the listening end. In the process of sending the handwriting update message, the handwriting update message can be distinguished by, for example, a sequence number and the order of each handwriting update message can be indicated. Generally speaking, the handwriting update message sent later is more updated, and the sequence number of the more updated handwriting update message can be marked larger. The main speaker or the listening end will store the current handwriting update message and its sequence number (called "current sequence number"). When the main speaker or the listening end obtains the handwriting information from the remote server after receiving the handwriting update message, the local (main speaker or the listening end) state needs to be considered. The local state includes an idle state and an acquisition state, wherein the acquisition state indicates that the main speaker or the listening end is obtaining the handwriting information set corresponding to the current sequence number from the remote server, and the idle state indicates that the main speaker or the listening end is not obtaining the handwriting information set corresponding to the current sequence number from the remote server, and is in a state of waiting for update.

[0137] like Figure 5 As shown, the main speaker or the listening end receives the MQTT message, obtains the handwriting sequence number and sets it as the MQTT message sequence number, and then checks whether the main speaker or the listening end is in an idle state.

[0138] When it is not in the idle state, that is, in the acquiring state, no matter whether the MQTT message sequence number is greater than the current sequence number (that is, no matter whether the MQTT message is a message that is newer than the update message indicated by the current sequence number), the annotation information set corresponding to the current handwriting update message is obtained from the remote server; after the annotation information set data is acquired, the state is set to the idle state, and the data with the current sequence number as the starting point and the MQTT message sequence number as the end point is acquired to determine whether the starting point is greater than or equal to the end point. In this case, if the starting point is less than the end point, the state is set to the acquiring state, and the handwriting information set corresponding to the MQTT message (sequence number) is acquired from the remote server.

[0139] When in idle state, obtain data with the current sequence number as the starting point and the MQTT message sequence number as the end point, and determine whether the starting point is greater than or equal to the end point. When the starting point is less than the end point, set the state to the acquisition state, and obtain the handwriting information set corresponding to the MQTT message sequence number from the remote server; when the starting point is greater than or equal to the end point, continue to be in idle state, and end the acquisition process corresponding to the MQTT message.

[0140] exist Figure 5In the process, whether obtaining the annotation information set corresponding to the current handwriting update message or the handwriting information set corresponding to the MQTT message from the remote server, when the data is successfully obtained, the obtained data (handwriting information set) is parsed, the drawn node list information is obtained, the handwriting point information in the node list information is added to the drawn handwriting, the local cache is updated, and the main speaker or the listening end updates the displayed handwriting according to the updated local cache.

[0141] exist Figure 5 In the process, whether obtaining the annotation information set corresponding to the current handwriting update message or obtaining the handwriting information set corresponding to the MQTT message from the remote server, when the data acquisition is unsuccessful, the acquisition process is terminated.

[0142] Thus, according to Figure 5 In the process shown, the acquisition subunit 7031 includes the following subunits:

[0143] The first acquisition branch unit 70311 is used for acquiring, in response to being in the acquisition state, a set of annotation information corresponding to the current handwriting update message from the remote server; and

[0144] The second acquisition branch unit 70312 is used to respond to being in an idle state. When the updated handwriting message from the remote server is an updated handwriting update message, the annotation information set corresponding to the updated handwriting message from the remote server is obtained from the remote server; otherwise, the annotation information set corresponding to the current handwriting update message is obtained from the remote server.

[0145] After receiving the annotation information set from the remote server, the speaker and the listener need to process the handwriting information set. The processing includes:

[0146] 1. Whether the annotation information set obtained from the remote server belongs to the division area where the current speaker or listener is located.

[0147] For example, in the application scenario of remote multi-terminal classrooms, the remote server can manage more than one remote classroom, and there may be a situation where the annotation information set of one remote classroom is sent to another remote classroom. Therefore, after receiving the annotation information set, the lecturer or the lecturer needs to determine whether the annotation information set belongs to the classroom to which the lecturer or the lecturer belongs. If it belongs to the classroom, the annotation information set is parsed; if it does not belong, the annotation information set is discarded.

[0148] 2. Determine whether the obtained annotation information set is empty.

[0149] For example, a specific algorithm can be used to determine whether the annotation information set is empty. If it is empty, no further parsing is required; if it is not empty, the internal data can be parsed.

[0150] 3. Obtain the display ratio data of the annotation information set and the display ratio data of the current speaker and listener, and display the handwriting information of the annotation information set after conversion.

[0151] Before displaying the handwriting information of the annotation information set, in order to correctly display the annotation handwriting, it is necessary to ensure that the display ratio of the received annotation information set is consistent with the display ratio of the current speaker end and the listener end.

[0152] The annotation generating unit 704 is used to generate annotation content on the canvas according to the local annotation information and the remote annotation information.

[0153] When new local annotation handwriting appears or new remote annotation handwriting is received, the speaker or listener needs to update the handwriting on the canvas.

[0154] When the annotation content is generated on the canvas according to the local annotation information, as described above, in one embodiment, when a lift event is detected, the handwriting corresponding to the entire handwriting path is drawn on the canvas. Figure 2 As shown, in the process of recording the coordinates of the point corresponding to the pressing event, the coordinates of the point corresponding to the moving event and the coordinates of the point corresponding to the lifting event in the handwriting path, the handwriting corresponding to the handwriting path is drawn on the canvas. In this way, it is possible to draw and display at the same time in the drawing mode.

[0155] Thus, the annotation generation unit 704 includes: a first annotation generation subunit 7041, which is used to draw handwriting corresponding to the handwriting path on the canvas in the process of recording the coordinates of the point corresponding to the pressing event, the coordinates of the point corresponding to the moving event and the coordinates of the point corresponding to the lifting event in the handwriting path.

[0156] When the annotation content is generated on the canvas according to the local annotation information, as described above, in one embodiment, when a lift event is detected, the handwriting that intersects with the clearing area is cleared on the canvas. In this way, when the lift event is triggered, the handwriting selected in the clearing area can be deleted.

[0157] Therefore, the annotation generating unit 704 includes: a second annotation generating sub-unit 7042, which is used to clear the handwriting intersecting with the clearing area on the canvas in response to the lifting event.

[0158] In the process of generating annotation content on the canvas according to the remote annotation information, it is necessary to generate annotation content on the canvas according to the handwriting object in the remote annotation information. In this process, it is necessary to use the node type information in the handwriting object to accurately draw the handwriting in the remote annotation information.

[0159] When new local annotation handwriting appears or new remote annotation handwriting is received, the speaker or listener needs to update the handwriting, which is mainly completed by traversing the handwriting list of the current user and other users in the local buffer. Figure 6 It is a flowchart of generating annotation content in the remote annotation method according to an embodiment of the present application.

[0160] like Figure 6 As shown, the process of generating annotation content includes: 1) determining whether there is a temporary bitmap. If there is no temporary bitmap, creating a new temporary bitmap and canvas; 2) clearing the canvas content. In one embodiment, the color of the canvas can be set to transparent; 3) traversing the handwriting objects corresponding to the local annotation information of the current user in the local cache, and generating annotation content on the canvas; 4) traversing the handwriting objects corresponding to the remote annotation information of other users in the local cache, and generating annotation content on the canvas; 5) drawing a temporary bitmap in the canvas, and displaying the annotation content generated by the canvas on the screen.

[0161] Therefore, the remote annotation device provided by the present application also includes:

[0162] The display unit 705 is used to display the annotation content.

[0163] According to drawing a temporary bitmap in the canvas, the annotation content generated by the canvas is displayed on the screen of the user end.

[0164] According to the remote annotation device provided by the present application, annotations in multiple remote terminals can be displayed to each other in real time, realizing the remote drawing and clearing functions of handwritings on multiple terminals, making the multi-terminal annotation process more interactive and improving the user experience.

[0165] See also Figure 8 , Figure 8 Provided is an electronic device, comprising a processor; and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the processor executes the computer instructions to implement the following Figure 1 The method and refinement scheme shown.

[0166] It should be understood that the above device embodiments are only illustrative, and the device disclosed in the present invention can also be implemented in other ways. For example, the division of units / modules described in the above embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0167] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present invention may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0168] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not otherwise specified, the processor or chip may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not otherwise specified, the on-chip cache, the off-chip memory, and the memory may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0169] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0170] The present application also provides a computer storage medium storing a computer program. When the computer program is executed by multiple processors, the processors execute the following steps: Figure 1 The method and refinement scheme shown.

[0171] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present application, the specific implementation methods and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as a limitation on the present application.

Claims

1. A remote annotation method, applied to a user end, comprising: Get local annotation information; Sending the local annotation information to a remote server; Acquiring remote annotation information from the remote server; Generating annotation content on the canvas according to the local annotation information and the remote annotation information; as well as Displaying the annotation content; The touch events corresponding to the local annotation information include a press event, a move event and a lift event, the acquiring of the local annotation information includes acquiring a handwriting object of the local annotation information, the handwriting object includes a drawing mode of the annotation handwriting, a handwriting path of the annotation handwriting and a clearing area of ​​the annotation handwriting, wherein the drawing mode includes a drawing mode and a clearing mode; When the drawing mode includes a clearing mode, obtaining local annotation information includes: In response to the pressing event, recording the coordinates of the point corresponding to the pressing event in the clearing area; In response to the movement event, recording the coordinates of the point corresponding to the movement event in the clearing area; and In response to the lifting event, recording the coordinates of a point corresponding to the lifting event in the clearing area; The sending of the local annotation information to a remote server comprises: In the process of recording the coordinates of the point corresponding to the pressing event, the coordinates of the point corresponding to the moving event and the coordinates of the point corresponding to the lifting event in the handwriting path, the handwriting path is sent to the remote server so that other terminals obtain the handwriting path through the remote server.

2. The method of claim 1, wherein: When the drawing mode includes a painting mode, obtaining local annotation information includes: In response to the pressing event, recording the coordinates of the point corresponding to the pressing event in the handwriting path; In response to the movement event, recording the coordinates of the point corresponding to the movement event in the handwriting path; and In response to the lift-up event, the coordinates of the point corresponding to the lift-up event are recorded in the handwriting path.

3. The method of claim 2, wherein: Generating annotation content on the canvas according to the local annotation information and the remote annotation information includes: In the process of recording the coordinates of the point corresponding to the pressing event, the coordinates of the point corresponding to the moving event and the coordinates of the point corresponding to the lifting event in the handwriting path, the handwriting corresponding to the handwriting path is drawn on the canvas.

4. The method of claim 1, wherein: Generating annotation content on the canvas according to the local annotation information and the remote annotation information includes: In response to the lift event, the handwriting that intersects with the clearing area is cleared on the canvas.

5. The method according to any one of claims 1 to 4, wherein: The handwriting object also includes node type information, and the acquisition of local annotation information also includes: The node types of the point corresponding to the pressing event, the point corresponding to the moving event and the point corresponding to the lifting event are respectively determined as a pressing point, a moving point and a lifting point, and recorded in the node type information.

6. The method according to any one of claims 1 to 4, wherein: The handwriting object further includes: a user identifier of the annotation handwriting, a handwriting identifier of the annotation handwriting, and pen setting parameters of the annotation handwriting.

7. The method according to any one of claims 1 to 4, wherein: The step of obtaining remote annotation information from a remote server includes: In response to the handwriting update message from the remote server, an annotation information set is obtained from the remote server.

8. The method of claim 7, wherein: The step of obtaining the annotation information set from the remote server in response to the handwriting update message from the remote server comprises: The response is in the acquisition state, and the annotation information set corresponding to the current handwriting update message is acquired from the remote server; The response is in an idle state. When the updated handwriting message from the remote server is an updated handwriting update message, the annotation information set corresponding to the updated handwriting message from the remote server is obtained from the remote server; otherwise, the annotation information set corresponding to the current handwriting update message is obtained from the remote server.

9. A remote annotation device, comprising: A first acquisition unit, used to acquire local annotation information; A sending unit, used for sending the local annotation information to a remote server; A second acquiring unit, configured to acquire remote annotation information from the remote server; An annotation generating unit, used for generating annotation content on the canvas according to the local annotation information and the remote annotation information; as well as A display unit, used for displaying the annotation content; The touch events corresponding to the local annotation information include a press event, a move event and a lift event, the acquiring of the local annotation information includes acquiring a handwriting object of the local annotation information, the handwriting object includes a drawing mode of the annotation handwriting, a handwriting path of the annotation handwriting and a clearing area of ​​the annotation handwriting, wherein the drawing mode includes a drawing mode and a clearing mode; When the drawing mode includes a clearing mode, the first acquiring unit includes: A fourth recording subunit, configured to record, in response to the pressing event, the coordinates of a point corresponding to the pressing event in the clearing area; A fifth recording subunit, configured to record, in response to the movement event, the coordinates of a point corresponding to the movement event in the clearing area; and a sixth recording subunit, configured to record, in response to the lifting event, the coordinates of a point corresponding to the lifting event in the clearing area; The sending unit is used for: In the process of recording the coordinates of the point corresponding to the pressing event, the coordinates of the point corresponding to the moving event and the coordinates of the point corresponding to the lifting event in the handwriting path, the handwriting path is sent to the remote server so that other terminals obtain the handwriting path through the remote server.

10. An electronic device comprising: processor; as well as A memory storing a computer program, which, when executed by the processor, enables the processor to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 8.

Images