Document annotation method and device for electronic whiteboard, equipment and storage medium

Abstract

The invention provides a document annotation method and device for an electronic whiteboard, equipment and a storage medium, and the document annotation method for the electronic whiteboard comprises the steps: obtaining voice information and touch information under the condition that the electronic whiteboard displays a document, judging whether to enter an annotation mode or not according to the voice information and the touch information, and carrying out the annotation of the document in the annotation mode. A user is prevented from mistakenly entering an annotation mode during normal explanation and writing, the annotation position is determined, the touch pressure value, the moving speed of a touch gesture, the contact area and the direction angle are obtained when it is determined that the annotation mode is entered, comprehensive analysis is conducted through the intention recognition model, the target operation intention is determined, the recognition accuracy of the target operation intention is improved, and the user experience is improved. The target operation is triggered according to the target operation intention, the annotation mode is provided through the target operation including the annotation content input operation, the zooming operation or the page turning operation, the situation that document content is shielded by annotation content input is avoided, meanwhile, the annotation mode further supports the zooming operation and the page turning operation, and the explanation effect is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of electronic whiteboards, and in particular to a document annotation method, device, equipment and storage medium for an electronic whiteboard. Background Art

[0002] With the development of electronic whiteboard technology, electronic whiteboards are gradually being used in various fields. For example, in the field of education, electronic whiteboards can provide teaching assistance for teachers. Teachers can open the corresponding courseware to explain knowledge to students. When explaining certain knowledge points, teachers may need to write out detailed steps in the corresponding locations for detailed explanation. However, when writing directly in writing mode, the written content can easily cover the original content in the courseware, causing the courseware content and the written content to overlap, which is not conducive to students' viewing and affects the effectiveness of the explanation. Summary of the Invention

[0003] The present invention provides a document annotation method, apparatus, device, and storage medium for an electronic whiteboard to solve at least one problem in the related art. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a method for annotating a document on an electronic whiteboard, comprising:

[0005] When the electronic whiteboard displays a document, voice information and touch information are obtained;

[0006] Determine whether to enter the annotation mode based on the voice information and touch information, and determine the annotation position when entering the annotation mode;

[0007] After the annotation position is determined, the touch pressure value, the movement speed, the contact area, and the direction angle of the touch gesture are obtained, and the touch pressure value, the movement speed, the contact area, and the direction angle are comprehensively analyzed through the intention recognition model to determine the target operation intention;

[0008] According to the target operation intention, the target operation is triggered, and the target operation includes annotation content input operation, zoom operation or page turning operation.

[0009] In one embodiment, judging whether to enter the annotation mode based on the voice information and the touch information, and determining the annotation position when entering the annotation mode comprises:

[0010] Perform speech recognition on the speech information to obtain speech text, and then segment the speech text to determine whether the segmented words contain preset key words;

[0011] When the preset keywords are included, the annotation mode is entered, and when entering the annotation mode, the annotation position is determined according to the touch information.

[0012] In one embodiment, when entering the annotation mode, determining the annotation position according to the touch information includes:

[0013] When entering the annotation mode, the touch information is analyzed to determine the path of the touch track;

[0014] When the path is within the document, the location of the content corresponding to the path in the document is used as the annotation location;

[0015] When the passing position is outside the document, the user is prompted that the annotation mode has been entered, and the annotation position is determined in response to the user's selection information of the content in the document.

[0016] In one embodiment, the performing of a comprehensive analysis of the touch pressure value, movement speed, contact area, and direction angle using an intention recognition model to determine the target operation intention includes:

[0017] Normalizing the touch pressure value, movement speed, contact area, and direction angle to obtain a standard touch pressure value, standard movement speed, standard contact area, and standard direction angle, and outputting a pressure feature vector for the standard touch pressure value, a speed feature vector for the standard movement speed, an area feature vector for the standard contact area, and a direction angle feature vector for the standard direction angle through a fully connected layer;

[0018] Through the attention mechanism, the pressure eigenvector, velocity eigenvector, area eigenvector and direction angle eigenvector are analyzed to determine the dynamic pressure weight corresponding to the pressure eigenvector, the dynamic velocity weight corresponding to the velocity eigenvector, the dynamic area weight corresponding to the area eigenvector and the dynamic direction angle weight corresponding to the direction angle eigenvector;

[0019] Perform weighted fusion according to the pressure eigenvector, dynamic pressure weight, velocity eigenvector, dynamic velocity weight, area eigenvector, dynamic area weight, direction angle eigenvector and dynamic direction angle weight to obtain a fused eigenvector;

[0020] Based on the fused feature vector and LSTM classifier, the probabilities corresponding to the annotation content input operation intention, zoom operation intention, and page turning operation intention are determined, and the operation intention corresponding to the largest probability is taken as the target operation intention.

[0021] In one embodiment, the method further comprises:

[0022] The maximum probability is used as a measure of confidence;

[0023] Determine the feedback vibration intensity based on the measured confidence and the current tactile gain coefficient;

[0024] Generate vibration feedback according to the feedback vibration intensity;

[0025] Among them, the feedback vibration intensity is positively correlated with the accuracy of the currently determined target operation intention.

[0026] In one embodiment, determining the feedback vibration intensity based on the measured confidence and the current tactile gain coefficient includes:

[0027] When the measured confidence is less than or equal to the confidence threshold, the feedback vibration intensity is determined to be zero;

[0028] When the measurement confidence is greater than the confidence threshold, a difference between the measurement confidence and the confidence threshold is determined, and the feedback vibration intensity is determined according to a first product of the difference and a current tactile gain coefficient.

[0029] In one embodiment, the current haptic gain coefficient is determined by:

[0030] Obtain historical touch data in annotation mode, analyze the historical touch data in annotation mode, determine the number of satisfactory vibration intensities reported by users under different tactile gain coefficients and the corresponding satisfaction scores for each satisfactory response, and calculate the satisfaction rate based on the number of satisfactory responses and the total number of vibrations;

[0031] Determine the second product of each satisfaction score and the score weight factor under different tactile gain coefficients, and determine the vibration comprehensive score under different tactile gain coefficients according to the sum of the second product and the satisfaction rate;

[0032] The tactile gain coefficient corresponding to the maximum vibration comprehensive score is used as the current tactile gain coefficient.

[0033] In a second aspect, an embodiment of the present application provides a document annotation device for an electronic whiteboard, comprising:

[0034] An acquisition module, used to acquire voice information and touch information when the electronic whiteboard displays a document;

[0035] A determination module, configured to determine whether to enter the annotation mode based on the voice information and the touch information, and to determine the annotation position when entering the annotation mode;

[0036] The intention module is used to obtain the touch pressure value, the movement speed, the contact area and the direction angle of the touch gesture after determining the annotation position, and to perform a comprehensive analysis of the touch pressure value, the movement speed, the contact area and the direction angle through the intention recognition model to determine the target operation intention;

[0037] The operation module is used to trigger the target operation according to the target operation intention. The target operation includes the annotation content input operation, the zoom operation or the page turning operation.

[0038] In one embodiment, the operation module is further configured to:

[0039] The maximum probability is used as a measure of confidence;

[0040] Determine the feedback vibration intensity based on the measured confidence and the current tactile gain coefficient;

[0041] Generate vibration feedback according to the feedback vibration intensity;

[0042] Among them, the feedback vibration intensity is positively correlated with the accuracy of the currently determined target operation intention.

[0043] In one embodiment, the operation module is further configured to:

[0044] Obtain historical touch data in annotation mode, analyze the historical touch data in annotation mode, determine the number of satisfactory vibration intensities reported by users under different tactile gain coefficients and the corresponding satisfaction scores for each satisfactory response, and calculate the satisfaction rate based on the number of satisfactory responses and the total number of vibrations;

[0045] Determine the second product of each satisfaction score and the score weight factor under different tactile gain coefficients, and determine the vibration comprehensive score under different tactile gain coefficients according to the sum of the second product and the satisfaction rate;

[0046] The tactile gain coefficient corresponding to the maximum vibration comprehensive score is used as the current tactile gain coefficient.

[0047] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method in any one of the above-mentioned embodiments.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, it implements the method in any one of the above-mentioned embodiments.

[0049] The beneficial effects of the above technical solution include at least:

[0050] By obtaining voice information and touch information when displaying a document on an electronic whiteboard, it is determined whether to enter the annotation mode based on the voice information and touch information, thereby preventing the user from accidentally entering the annotation mode during normal explanations and writing. When determining to enter the annotation mode, the annotation position is determined, the touch pressure value, the movement speed, the contact area and the direction angle of the touch gesture are obtained, and the touch pressure value, the movement speed, the contact area and the direction angle are comprehensively analyzed through the intention recognition model to determine the target operation intention, improve the recognition accuracy of the target operation intention, trigger the target operation according to the target operation intention, and provide the annotation mode through the target operations including the annotation content input operation, the zoom operation or the page turning operation to avoid the input of the annotation content blocking the document content. At the same time, the annotation mode also supports the zoom operation and the page turning operation, which is conducive to ensuring the explanation effect.

[0051] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0053] Figure 1 This is a flowchart of the steps of a document annotation method for an electronic whiteboard according to an embodiment of the present application;

[0054] Figure 2 This is a structural block diagram of a document annotation device for an electronic whiteboard according to an embodiment of the present application;

[0055] Figure 3 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0057] Reference Figure 1 , showing a flowchart of a document annotation method for an electronic whiteboard according to an embodiment of the present application, the document annotation method for an electronic whiteboard may include at least steps S100-S400:

[0058] S100 : When the electronic whiteboard displays a document, obtain voice information and touch information.

[0059] S200: Determine whether to enter the annotation mode according to the voice information and the touch information, and determine the annotation position when entering the annotation mode.

[0060] S300. After determining the annotation position, obtain the touch pressure value, the moving speed, the contact area and the direction angle of the touch gesture, and perform a comprehensive analysis of the touch pressure value, the moving speed, the contact area and the direction angle through the intention recognition model to determine the target operation intention.

[0061] S400: triggering a target operation according to the target operation intention.

[0062] Optionally, the target operation includes but is not limited to an annotation content input operation, a zoom operation, a page turning operation, or no operation (eg, an accidental touch).

[0063] The document annotation method of the electronic whiteboard in the embodiment of the present application can be executed by the processor of the electronic whiteboard.

[0064] The technical solution of the embodiment of the present application obtains voice information and touch information when a document is displayed on an electronic whiteboard, and determines whether to enter the annotation mode based on the voice information and touch information, so as to avoid the user from accidentally entering the annotation mode during normal explanation and writing. When it is determined that the annotation mode has been entered, the annotation position is determined, the touch pressure value, the movement speed, the contact area and the direction angle of the touch gesture are obtained, and the touch pressure value, the movement speed, the contact area and the direction angle are comprehensively analyzed through the intention recognition model to determine the target operation intention, improve the recognition accuracy of the target operation intention, trigger the target operation according to the target operation intention, and provide the annotation mode through the target operation including the annotation content input operation, the zoom operation or the page turning operation, so as to avoid the input of the annotation content blocking the document content. At the same time, the annotation mode also supports the zoom operation and the page turning operation, which is conducive to ensuring the explanation effect.

[0065] In one embodiment, the electronic whiteboard supports importing multiple document types, including but not limited to PDF, PPT, and WORD. Users can open documents and display them on the electronic whiteboard. When the electronic whiteboard displays documents, user voice information and touch information are collected. Touch information is information generated by the user touching the electronic whiteboard screen, including but not limited to the coordinates of at least one touch point and the time when the touch point coordinates were generated.

[0066] In one embodiment, since the user will explain the course through voice during the class, it is necessary to recognize the voice information to determine whether the current mode is to explain the course or to trigger the annotation mode through voice wake-up. Optionally, step S200 includes steps S210-S220:

[0067] S210: Perform speech recognition on the speech information to obtain speech text, and perform word segmentation on the speech text to determine whether the words obtained by the word segmentation contain preset key words.

[0068] Optionally, natural language processing technology can be used to perform speech recognition on the voice information to obtain voice text. Then, the voice text is segmented using a word segmentation algorithm to obtain several words, which are then compared to determine whether the words obtained by segmentation contain preset key words. For example, the user can preset key words to wake up the annotation mode in advance, such as "start annotation" and "enter annotation", and then compare to determine whether it contains the key words "start" and "annotation" or whether it contains "enter" and "annotation".

[0069] S220: Entering into the annotation mode when a preset keyword is contained, and determining the annotation position according to the touch information when entering into the annotation mode.

[0070] Optionally, if the segmented words contain the above-mentioned preset key words, the electronic whiteboard will enter the annotation mode. When entering the annotation mode, the annotation position is determined according to the touch information, including S2201-S2203:

[0071] S2201: When entering the annotation mode, analyze the touch information to determine the path of the touch track.

[0072] Optionally, when entering the annotation mode, the processor of the electronic whiteboard analyzes the touch information. Since the touch information may contain one or more touch point coordinates and the generation time of the touch point coordinates, by sorting the generation time, it is possible to know the path and time of the corresponding touch trajectory.

[0073] S2202: When the passing path is located in the document, the location of the content corresponding to the passing path in the document is used as the annotation location.

[0074] Optionally, when the passing path is within the document, for example, when the user speaks a voice message including "start annotating", the user touches the location of the document on the screen, and it can be known that the user needs to annotate the document. Therefore, at this time, the location of the content corresponding to the passing path in the document can be used as the annotation location. For example, if the passing path is a circle, and the circle circles the word "trees", then the location of the content "trees" corresponding to the passing path of the circle is determined, and the location is used as the annotation location. It should be noted that the passing path can be other graphics without limitation, and in addition to the circled method, it can also be a line method; if the document is a PPT or PDF, the annotation form generated at the annotation location can be a comment box, and when the annotation content is subsequently entered, it is entered in the form of a comment box. If the document is a WORD, the annotation form generated at the annotation location can be a comment column, and when the annotation content is subsequently entered at the annotation location, it is entered in the form of a comment column, and there is no specific limitation.

[0075] S2203: When the passing position is outside the document, the user is prompted that the annotation mode has been entered, and the annotation position is determined in response to the user's selection information of the content in the document.

[0076] Optionally, when the passing position is outside the document, the user may have just woken up and entered the annotation mode but has not touched the content in the document in time. The processor of the electronic whiteboard can remind the user that he has entered the annotation mode by sending voice or pop-up window, so that the user can input the selected information in time (for example, a selected text, a word, a sentence, etc.). The processor responds to the user's selected information on the content in the document, and determines the location of a text, a word or a sentence, etc. selected based on the selected information as the annotation position.

[0077] In one embodiment, when the annotation position is determined, the user can generate the annotation content by writing, and then write and display it in the above-mentioned annotation form. Therefore, when writing, the electronic whiteboard can collect the touch pressure value, the moving speed of the touch gesture, the contact area and the direction angle. It should be noted that, assuming that the user involves M touch points when writing, each touch point will have a pressure value, and the average value of the M touch points is calculated as the touch pressure value; the instantaneous speed can be determined based on the displacement per unit time and the ratio per unit time, and the moving speed of the touch gesture can be obtained by calculating the average instantaneous speed based on the instantaneous speed involved in the M touch points; the contact area can calculate the area of the touch area formed by the M touch points; the ratio between the vertical coordinate change and the horizontal coordinate change between two touch points arranged in time order determines the sub-direction angle, and the sum of all sub-direction angles is used as the final direction angle.

[0078] Then, in S300, the touch pressure value, movement speed, contact area and direction angle are comprehensively analyzed through the intention recognition model to determine the target operation intention, including: S310-S340:

[0079] S310. Normalize the touch pressure value, movement speed, contact area, and direction angle respectively to obtain a standard touch pressure value, a standard movement speed, a standard contact area, and a standard direction angle, and output a pressure feature vector of the standard touch pressure value, a speed feature vector of the standard movement speed, an area feature vector of the standard contact area, and a direction angle feature vector of the standard direction angle through a fully connected layer.

[0080] Optionally, firstly, a normalization function is used to normalize the touch pressure value, movement speed, contact area and direction angle respectively to obtain a standard touch pressure value, a standard movement speed, a standard contact area and a standard direction angle. In the embodiment of the present application, x′ i For example, when i takes values of 1, 2, 3, or 4, it can correspond to the standard touch pressure value, standard movement speed, standard contact area, and standard direction angle, so that the standard touch pressure value, standard movement speed, standard contact area, and standard direction angle are ultimately within the same range.

[0081] Then, the fully connected layer is used to output the pressure feature vector of the standard touch pressure value, the speed feature vector of the standard moving speed, the area feature vector of the standard contact area, and the direction angle feature vector f(x′) of the standard direction angle. i ), when i takes values of 1, 2, 3, or 4, the pressure eigenvector, velocity eigenvector, area eigenvector, and azimuth eigenvector can be obtained.

[0082] S320. Analyze the pressure eigenvector, velocity eigenvector, area eigenvector, and azimuth eigenvector through the attention mechanism to determine the dynamic pressure weight corresponding to the pressure eigenvector, the dynamic velocity weight corresponding to the velocity eigenvector, the dynamic area weight corresponding to the area eigenvector, and the dynamic azimuth weight of the azimuth eigenvector.

[0083] Optionally, the pressure eigenvector, velocity eigenvector, area eigenvector, and azimuth eigenvector are analyzed through an attention mechanism to determine the dynamic pressure weight corresponding to the pressure eigenvector, the dynamic velocity weight corresponding to the velocity eigenvector, the dynamic area weight corresponding to the area eigenvector, and the dynamic azimuth weight of the azimuth eigenvector:

[0084]

[0085] Among them, w iWhen i is 1, 2, 3, or 4, the dynamic pressure weight, dynamic velocity weight, dynamic area weight, and dynamic direction angle weight can be obtained. n = i-1, x′ j Represents x′ i One of the other three eigenvectors is different, so the denominator is equivalent to calculating x′ i The sum of the results obtained by using the exponential function for the other three different eigenvectors.

[0086] S330 , performing weighted fusion according to the pressure eigenvector, dynamic pressure weight, velocity eigenvector, dynamic velocity weight, area eigenvector, dynamic area weight, azimuth eigenvector, and dynamic azimuth weight to obtain a fused eigenvector.

[0087] Optionally, after determining each eigenvector and the dynamic weight corresponding to each eigenvector, weighted fusion is performed based on the pressure eigenvector, dynamic pressure weight, velocity eigenvector, dynamic velocity weight, area eigenvector, dynamic area weight, directional angle eigenvector and dynamic directional angle weight to obtain a fused eigenvector for comprehensively expressing the touch content of each dimension.

[0088] S340. Determine the probabilities corresponding to the annotation content input operation intention, the zoom operation intention, and the page turning operation intention based on the fused feature vector and the LSTM classifier, and take the operation intention corresponding to the largest probability as the target operation intention.

[0089] Optionally, the fused feature vector is input into the LSTM classifier, at which time the LSTM classifier will output the probability distribution of each operation intention (P1, P2, P3, P4), where P1 is the probability corresponding to the annotation content input operation intention, P2 is the probability corresponding to the zoom operation intention, P3 is the probability corresponding to the page turning operation intention, and P4 is the probability corresponding to no operation (i.e., the probability that the processor believes that the current touch is an erroneous operation). In some implementations, there may be no P4. Finally, the operation intention corresponding to the largest probability is used as the target operation intention. For example, if P1 is the largest, the annotation content input operation intention is determined to be the target operation intention, and if P3 is the largest, the annotation content input operation intention is determined to be the page turning operation intention.

[0090] In addition, the attention mechanism, LSTM classifier, etc. included in the intent recognition model are obtained by collecting a large number of touch samples (including touch samples of the above-mentioned operations) for training and adjusting hyperparameters in advance.

[0091] It should be noted that, in the embodiment of the present application, in order to ensure the user's teaching experience, even if the annotation mode is entered, the annotation mode also supports operations such as zooming and page turning in addition to the annotation content input operation, so that the user can perform different operations through different pre-set gestures, without having to exit the annotation mode in order to use other operations. Otherwise, there will be multiple entry and recognition of the annotation mode, which is very cumbersome. Accordingly, since the annotation mode supports multiple operations, in order to correctly identify what the current operation intention is, since the touch pressure, movement speed, contact area, and direction angle of different gestures will be different, these factors are subjected to a multi-factor comprehensive analysis to output the probability distribution (P1, P2, P3, P4) of each operation intention, which is conducive to improving the accuracy of intention recognition.

[0092] In one embodiment, in step S400, the target operation is triggered according to the target operation intention. For example, the target operation includes but is not limited to the annotation content input operation, the zoom operation, the page turning operation and the no operation (i.e., it is considered to be an accidental touch). In other embodiments, there may be no no operation. For example, if the target operation intention is the annotation content input operation intention, triggering the target operation is the annotation content input operation, and adding the annotation content to the annotation form (annotation column or annotation box) so that the content input by the user will not block the content in the document in the annotation form, so as to ensure the explanation effect; if the target operation intention is the page turning operation intention, triggering the target operation is triggering the page turning operation to realize the page turning of the document.

[0093] In one embodiment, the document annotation method of the electronic whiteboard of the embodiment of the present application may further include steps S510-S530:

[0094] S510. Use the maximum probability as a measure of confidence.

[0095] Optionally, max(P1, P2, P3, P4) is used as a measure of confidence c. For example, if P3 is the largest, then c=P3.

[0096] S520 : Determine the feedback vibration intensity according to the measured confidence and the current tactile gain coefficient.

[0097] Specifically, the confidence threshold is 0.5 as an example. When the measurement confidence c is less than or equal to 0.5, the feedback vibration intensity is determined to be zero, which indicates that the measurement confidence c is low at this time, and the processor has "no confidence" in the current target operation intention, and believes that the accuracy is low; when the measurement confidence c is greater than the confidence threshold 0.5, it is considered that the accuracy is high, and the difference between the measurement confidence and the confidence threshold (c-0.5) is determined. The feedback vibration intensity I is determined based on the first product of the difference and the current tactile gain coefficient k:

[0098] I=(c-0.5)×k

[0099] Therefore, as can be seen from the above formula, since the confidence level is used to characterize the accuracy of the target operation intention, the feedback vibration intensity I is positively correlated with the accuracy of the currently determined target operation intention, and the tactile gain coefficient k is also positively correlated with the feedback vibration intensity I. It is understandable that the user can understand the accuracy of the current processor's recognition of the target operation intention through the strength of the feedback vibration intensity they feel. Therefore, based on this principle, in some cases, such as when the probability of the target operation intention is low (for example, 0.55), but the target operation intention is actually accurately recognized, a reasonable tactile gain coefficient k can be determined to maximize the feedback vibration intensity I and provide the user with a better feedback feel.

[0100] In one embodiment, the current tactile gain coefficient k is determined by:

[0101] 1. Obtain historical touch data in annotation mode, analyze the historical touch data in annotation mode, determine the number of times users are satisfied with the vibration intensity under different tactile gain coefficients and the corresponding satisfaction score for each time they are satisfied, and calculate the satisfaction rate based on the number of satisfaction times and the total number of vibrations.

[0102] It should be noted that the processor will randomly change the value of the tactile gain coefficient k at preset intervals, thereby obtaining touch data under different tactile gain coefficients k, forming historical touch data in the annotation mode. Among them, the user can turn on the feedback option in the system settings, so that every time the user performs a touch operation in the annotation mode, the processor will identify the target operation intention and ask the user whether it is correct. If the feedback is correct, it will further ask whether the user is satisfied with the vibration intensity of the current feedback. If satisfied, the number of satisfactions will be recorded once, and after confirming satisfaction, the satisfaction score will be asked (for example, 0.1, 0.2, 0.3...1 point). In this way, the number of satisfactions of the vibration intensity of user feedback under different tactile gain coefficients and the corresponding satisfaction score M(k) for each time of satisfaction can be counted, and then the satisfaction rate S(k) is calculated based on the ratio of the number of satisfactions to the total number of vibrations.

[0103] 2. Determine the second product of each satisfaction score M(k) and the score weight factor β (preset based on experience, not specifically limited, for example, 0.1-0.3) under different tactile gain coefficients k, and determine the vibration comprehensive score J(k) under different tactile gain coefficients based on the sum of the second product and the satisfaction rate S(k):

[0104] J(k)=M(k)×β+S(k)

[0105] 3. The tactile gain coefficient corresponding to the maximum vibration comprehensive score J(k) is used as the current tactile gain coefficient.

[0106] S530: Generate vibration feedback according to the feedback vibration intensity.

[0107] Optionally, after the feedback vibration intensity I is obtained, a vibration feedback with a vibration intensity of the feedback vibration intensity I is generated for the user.

[0108] Reference Figure 2 , shows a structural block diagram of a document annotation device for an electronic whiteboard according to an embodiment of the present application, the device may include:

[0109] An acquisition module, used to acquire voice information and touch information when the electronic whiteboard displays a document;

[0110] A determination module, configured to determine whether to enter the annotation mode based on the voice information and the touch information, and to determine the annotation position when entering the annotation mode;

[0111] The intention module is used to obtain the touch pressure value, the movement speed, the contact area and the direction angle of the touch gesture after determining the annotation position, and to perform a comprehensive analysis of the touch pressure value, the movement speed, the contact area and the direction angle through the intention recognition model to determine the target operation intention;

[0112] The operation module is used to trigger the target operation according to the target operation intention. The target operation includes the annotation content input operation, the zoom operation or the page turning operation.

[0113] In one embodiment, the operation module is further configured to:

[0114] The maximum probability is used as a measure of confidence;

[0115] Determine the feedback vibration intensity based on the measured confidence and the current tactile gain coefficient;

[0116] Generate vibration feedback according to the feedback vibration intensity;

[0117] Among them, the feedback vibration intensity is positively correlated with the accuracy of the currently determined target operation intention.

[0118] In one embodiment, the operation module is further configured to:

[0119] Obtain historical touch data in annotation mode, analyze the historical touch data in annotation mode, determine the number of satisfactory vibration intensities reported by users under different tactile gain coefficients and the corresponding satisfaction scores for each satisfactory response, and calculate the satisfaction rate based on the number of satisfactory responses and the total number of vibrations;

[0120] Determine the second product of each satisfaction score and the score weight factor under different tactile gain coefficients, and determine the vibration comprehensive score under different tactile gain coefficients according to the sum of the second product and the satisfaction rate;

[0121] The tactile gain coefficient corresponding to the maximum vibration comprehensive score is used as the current tactile gain coefficient.

[0122] The functions of each module in the device of the embodiment of the present application can be found in the corresponding description in the above method and will not be repeated here.

[0123] Reference Figure 3 , shows a block diagram of an electronic device according to an embodiment of the present application. The electronic device includes a memory 310 and a processor 320. The memory 310 stores instructions executable on the processor 320. The processor 320 loads and executes the instructions to implement the document annotation method for the electronic whiteboard according to the above embodiment. The number of the memory 310 and the processor 320 can be one or more.

[0124] In one embodiment, the electronic device further includes a communication interface 330 for communicating with external devices and performing data exchange transmission. If the memory 310, the processor 320, and the communication interface 330 are implemented independently, the memory 310, the processor 320, and the communication interface 330 can be interconnected through a bus and complete communication with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0125] Optionally, in a specific implementation, if the memory 310, the processor 320 and the communication interface 330 are integrated on a chip, the memory 310, the processor 320 and the communication interface 330 can communicate with each other through an internal interface.

[0126] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the document annotation method for an electronic whiteboard provided in the above embodiment.

[0127] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided in the embodiment of the present application.

[0128] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0129] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the advanced reduced instruction set machine (ARM) architecture.

[0130] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0131] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0132] In the description of this specification, the reference terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0133] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, features specified as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0134] Any process or method description in a flow chart or otherwise described herein can be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present application includes additional implementations in which the functions may be performed in a different order than shown or discussed, including in a substantially simultaneous manner or in a reverse order depending on the functions involved.

[0135] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0136] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0137] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0138] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A document annotation method for an electronic whiteboard, characterized in that: include: When the electronic whiteboard displays a document, voice information and touch information are obtained; Determine whether to enter the annotation mode based on the voice information and touch information, and determine the annotation position when entering the annotation mode; After the annotation position is determined, the touch pressure value, the movement speed, the contact area, and the direction angle of the touch gesture are obtained, and the touch pressure value, the movement speed, the contact area, and the direction angle are comprehensively analyzed through the intention recognition model to determine the target operation intention; According to the target operation intention, the target operation is triggered, and the target operation includes annotation content input operation, zoom operation or page turning operation.

2. The document annotation method for an electronic whiteboard according to claim 1, characterized in that: The determining whether to enter the annotation mode according to the voice information and the touch information, and determining the annotation position when entering the annotation mode comprises: Perform speech recognition on the speech information to obtain speech text, and then segment the speech text to determine whether the segmented words contain preset key words; When the preset keywords are included, the annotation mode is entered, and when entering the annotation mode, the annotation position is determined according to the touch information.

3. The document annotation method for an electronic whiteboard according to claim 2, characterized in that: When entering the annotation mode, determining the annotation position according to the touch information includes: When entering the annotation mode, the touch information is analyzed to determine the path of the touch track; When the path is within the document, the location of the content corresponding to the path in the document is used as the annotation location; When the passing position is outside the document, the user is prompted that the annotation mode has been entered, and the annotation position is determined in response to the user's selection information of the content in the document.

4. The document annotation method for an electronic whiteboard according to any one of claims 1 to 3, characterized in that: The intention recognition model comprehensively analyzes the touch pressure value, movement speed, contact area, and direction angle to determine the target operation intention, including: Normalizing the touch pressure value, movement speed, contact area, and direction angle to obtain a standard touch pressure value, standard movement speed, standard contact area, and standard direction angle, and outputting a pressure feature vector for the standard touch pressure value, a speed feature vector for the standard movement speed, an area feature vector for the standard contact area, and a direction angle feature vector for the standard direction angle through a fully connected layer; Through the attention mechanism, the pressure eigenvector, velocity eigenvector, area eigenvector and direction angle eigenvector are analyzed to determine the dynamic pressure weight corresponding to the pressure eigenvector, the dynamic velocity weight corresponding to the velocity eigenvector, the dynamic area weight corresponding to the area eigenvector and the dynamic direction angle weight corresponding to the direction angle eigenvector; Perform weighted fusion according to the pressure eigenvector, dynamic pressure weight, velocity eigenvector, dynamic velocity weight, area eigenvector, dynamic area weight, direction angle eigenvector and dynamic direction angle weight to obtain a fused eigenvector; Based on the fused feature vector and LSTM classifier, the probabilities corresponding to the annotation content input operation intention, zoom operation intention, and page turning operation intention are determined, and the operation intention corresponding to the largest probability is taken as the target operation intention.

5. The document annotation method for an electronic whiteboard according to claim 4, characterized in that: The method further comprises: The maximum probability is used as a measure of confidence; Determine the feedback vibration intensity based on the measured confidence and the current tactile gain coefficient; Generate vibration feedback according to the feedback vibration intensity; Among them, the feedback vibration intensity is positively correlated with the accuracy of the currently determined target operation intention.

6. The document annotation method for an electronic whiteboard according to claim 5, characterized in that: Determining the feedback vibration intensity based on the measured confidence and the current tactile gain coefficient includes: When the measured confidence is less than or equal to the confidence threshold, the feedback vibration intensity is determined to be zero; When the measurement confidence is greater than the confidence threshold, a difference between the measurement confidence and the confidence threshold is determined, and the feedback vibration intensity is determined according to a first product of the difference and a current tactile gain coefficient.

7. The document annotation method for an electronic whiteboard according to claim 6, characterized in that: The current haptic gain factor is determined as follows: Obtain historical touch data in annotation mode, analyze the historical touch data in annotation mode, determine the number of satisfactory vibration intensities reported by users under different tactile gain coefficients and the corresponding satisfaction scores for each satisfactory response, and calculate the satisfaction rate based on the number of satisfactory responses and the total number of vibrations; Determine the second product of each satisfaction score and the score weight factor under different tactile gain coefficients, and determine the vibration comprehensive score under different tactile gain coefficients according to the sum of the second product and the satisfaction rate; The tactile gain coefficient corresponding to the maximum vibration comprehensive score is used as the current tactile gain coefficient.

8. A document annotation device for an electronic whiteboard, characterized in that: include: An acquisition module, used to acquire voice information and touch information when the electronic whiteboard displays a document; A determination module, configured to determine whether to enter the annotation mode based on the voice information and the touch information, and to determine the annotation position when entering the annotation mode; The intention module is used to obtain the touch pressure value, the movement speed, the contact area and the direction angle of the touch gesture after determining the annotation position, and to perform a comprehensive analysis of the touch pressure value, the movement speed, the contact area and the direction angle through the intention recognition model to determine the target operation intention; The operation module is used to trigger the target operation according to the target operation intention. The target operation includes the annotation content input operation, the zoom operation or the page turning operation.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images