Blackboard writing display method, electronic equipment and storage medium

Abstract

The invention provides a blackboard-writing display method, electronic equipment and a storage medium, the method is applied to the technical field of interactive tablets, and the method comprises the following steps: displaying a whiteboard application interface; in response to the identification instruction for the whiteboard application interface, identifying the original handwriting on the whiteboard application interface to obtain a handwriting identification result; acquiring voice information recorded in the writing process, and recognizing the voice information to obtain a voice recognition result; generating structured blackboard writing content according to the handwriting recognition result and the voice recognition result; wherein the writing style of the structured blackboard writing content is the same as that of the original handwriting; and displaying the structured blackboard writing content on the whiteboard application interface. According to the method, structuring of the blackboard-writing content is achieved, the readability, the constitutive property and the semantic integrity of the blackboard-writing content are improved, and a user can understand and reuse the blackboard-writing content conveniently.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of interactive whiteboard, and more particularly, to a display method of board writing, an electronic device and a storage medium in the technical field of interactive whiteboard. BACKGROUND

[0002] With the development of Internet technology, the scene of online meeting through terminal products such as interactive whiteboard is increasingly common, and users often communicate in various aspects such as work and study through meetings.

[0003] At present, the writing mode of board writing is often used in the meeting process, but in the writing process, only key words, pinyin or English abbreviations, ellipses, etc. are often written, the writing page is relatively messy, which leads to that the board content is difficult to understand after the meeting ends, and it is difficult to realize the meeting process based on the board content. SUMMARY

[0004] The present application provides a display method of board writing, an electronic device and a readable storage medium, which realizes the structuring of the board content, is beneficial to improving the readability, structure and semantic integrity of the board content, and is convenient for users to understand and reuse the board content.

[0005] In a first aspect, a display method of board writing is provided, which includes: displaying a whiteboard application interface; in response to a recognition instruction of the whiteboard application interface, recognizing original writing traces on the whiteboard application interface to obtain a handwriting recognition result; obtaining voice information recorded in the writing process and recognizing the voice information to obtain a voice recognition result; generating structured board content according to the handwriting recognition result and the voice recognition result; wherein the structured board content is of the same writing style as the original writing traces; and displaying the structured board content on the whiteboard application interface.

[0006] The technical solution described above fuses multi-dimensional input information (original writing handwriting and voice information), realizes semantic understanding and structured reconstruction of the board content, and does not stop at the visual level of beautification. Specifically, by recognizing the original writing handwriting, the voice information of the user in the writing process is recognized at the same time as a context supplement, which is beneficial to improve the accuracy of recognition and reduce the manual correction burden caused by OCR recognition errors. By combining the voice recognition result, it is beneficial to supplement the handwriting recognition result with semantics, fuse and associate the handwriting recognition result and the voice recognition result, so as to construct a structured board content that is clear in logic and complete in logic, make up for the semantic loss problem caused by relying on simple text alone, accurately restore the conference context after the conference, and facilitate subsequent review and continued discussion. Moreover, the structured board content retains the writing style of the original writing handwriting, meets the demand for clear logic, retains the user's personalized expression, and enhances the user's emotional recognition and use willingness.

[0007] In combination with the first aspect, in some possible implementation manners, displaying the structured board content on the whiteboard application interface includes: in a case where the original writing handwriting is retained on the whiteboard application interface, displaying the structured board content on the whiteboard application interface in a first display manner; and the first display manner is different from a display manner of the original writing handwriting on the whiteboard application interface.

[0008] In combination with the first aspect and the implementation manners described above, in some possible implementation manners, the first display manner is highlight display.

[0009] In combination with the first aspect and the implementation manners described above, in some possible implementation manners, the structured board content includes theme content corresponding to a plurality of theme labels respectively; and displaying the structured board content on the whiteboard application interface includes: displaying the plurality of theme labels on the whiteboard application interface; in response to a click operation on a target theme label, displaying theme content corresponding to the target theme label on the whiteboard application interface in a second display manner, and displaying theme content corresponding to non-target theme labels on the whiteboard application interface in a third display manner; and the target theme label is any one of the plurality of theme labels.

[0010] In combination with the first aspect and the implementation manners described above, in some possible implementation manners, the second display manner is highlight display, and the third display manner is non-highlight display.

[0011] In combination with the first aspect and the implementation manners described above, in some possible implementation manners, before displaying the plurality of theme labels on the whiteboard application interface, the method further includes: performing theme classification on the structured board content according to the handwriting recognition result and the voice recognition result, to obtain the plurality of theme labels.

[0012] With reference to the first aspect and the preceding implementation manners, in some possible implementation manners, the original writing handwriting includes original symbols, the handwriting recognition result includes semantic information of the original symbols, region features of a distribution of the original writing handwriting, and color information of the original writing handwriting; and the structured blackboard content is classified into themes according to the handwriting recognition result and the speech recognition result, to obtain a plurality of theme labels, including: extracting keywords based on the speech recognition result; and classifying the structured blackboard content into themes according to the semantic information of the original symbols, the region features, the color information, and the keywords, to obtain the plurality of theme labels.

[0013] With reference to the first aspect and the preceding implementation manners, in some possible implementation manners, the original writing handwriting includes original symbols and original characters; and the original writing handwriting on the whiteboard application interface is recognized to obtain the handwriting recognition result, including: recognizing the original symbols written by the user on the whiteboard application interface to obtain handwriting recognition results of the original symbols; and recognizing original characters in a preset range centered on the recognized original symbols to obtain handwriting recognition results of the original characters.

[0014] With reference to the first aspect and the preceding implementation manners, in some possible implementation manners, the structured blackboard content is generated according to the handwriting recognition result and the speech recognition result, including: correcting the original writing handwriting according to the speech recognition result and the handwriting recognition result to obtain target writing handwriting after correction; and generating the structured blackboard content according to the target writing handwriting after correction.

[0015] With reference to the first aspect and the preceding implementation manners, in some possible implementation manners, the speech recognition result includes semantic information of a plurality of speech segments, and the handwriting recognition result includes semantic information of a plurality of original writing handwritings; and the original writing handwriting is corrected according to the speech recognition result and the handwriting recognition result to obtain target writing handwriting after correction, including: determining time stamps of the plurality of original writing handwritings and determining time stamps of the plurality of speech segments; establishing a mapping relationship between the plurality of speech segments and the plurality of original writing handwritings according to the time stamps of the plurality of original writing handwritings and the time stamps of the plurality of speech segments; and for the speech segments and the original writing handwritings that have the mapping relationship, the original writing handwriting is corrected according to the semantic information of the speech segments and the semantic information of the original writing handwriting to obtain the target writing handwriting after correction.

[0016] In a possible implementation manner of the first aspect and the above implementation manners, according to the semantic information of the voice segment and the semantic information of the original writing trace, the original writing trace is modified, including: according to the semantic information of the voice segment and the semantic information of the original writing trace, one or any combination of the following steps is performed to modify the original writing trace: redundant writing traces existing in the original writing trace are removed; error writing traces existing in the original writing trace are corrected; writing traces to be beautified existing in the original writing trace are beautified; wherein the writing style of the beautified writing trace is the same as that of the un-beautified writing trace; omitted writing traces are supplemented.

[0017] In a second aspect, an electronic device is provided, including: a memory for storing executable program code; a processor for calling and running the executable program code from the memory, so that the electronic device executes the method in the first aspect or any possible implementation manner of the first aspect.

[0018] In a third aspect, a computer program product is provided, including: computer program code, when the computer program code is run on a computer, so that the computer executes the method in the first aspect or any possible implementation manner of the first aspect.

[0019] In a fourth aspect, a computer readable storage medium is provided, which stores computer program code, when the computer program code is run on a computer, so that the computer executes the method in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a display interface schematic diagram of an interactive tablet provided by an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of a whiteboard application interface provided by an embodiment of the present application;

[0022] Figure 3 is a schematic flow chart of a display method of board writing provided by an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a whiteboard application interface before an original writing trace is identified, provided by an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a whiteboard application interface in a process of identifying an original writing trace, provided by an embodiment of the present application;

[0025] Figure 6is a schematic diagram of a display interface of structured board writing content provided by an embodiment of the present application;

[0026] Figure 7 is a schematic flow chart of another display method of board writing provided by an embodiment of the present application;

[0027] Figure 8 is a schematic diagram of a whiteboard application interface with a theme label provided by an embodiment of the present application;

[0028] Figure 9 is a schematic diagram of another whiteboard application interface with a theme label provided by an embodiment of the present application;

[0029] Figure 10 is a schematic flow chart of another display method of board writing provided by an embodiment of the present application;

[0030] Figure 11 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the present application will be described in detail below with reference to the drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B: "and / or" in the text only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0032] Hereinafter, the terms "first" and "second" are used only for descriptive purposes and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more features.

[0033] With the development of Internet technology, the scene of online meeting through terminal products such as interactive tablets is becoming more and more common, and users often communicate in various aspects such as work and study through meetings.

[0034] During the meeting, board writing is a common and efficient way of communication. Users usually record key information such as keywords, abbreviations, symbols, pinyin, English expressions, etc. on the whiteboard application interface in the form of handwriting. However, due to the fast pace of the meeting and high information density, the actual board writing content is often brief and lacks complete semantic expression.

[0035] The inventors of this application discovered through research that related technologies primarily focus on visually enhancing the original handwriting, such as converting handwritten fonts into standard fonts like KaiTi or SongTi. While these methods improve the visual presentation of the blackboard content, they do not effectively process the content at the semantic and structural levels. This makes it difficult to recall the original intent after the meeting, especially when removed from the meeting context, where simplified expressions can easily lead to misunderstandings. Furthermore, current blackboard recognition methods based on OCR (Optical Character Recognition) technology have many limitations in practical applications. For example, OCR has low accuracy and a high false positive rate in recognizing cursive, cursive, and cross-language mixed writing (such as a mix of Chinese and English).

[0036] In summary, the relevant technologies have failed to effectively address the problem that whiteboard content is difficult to understand and difficult to continue discussion after the meeting.

[0037] To at least address the aforementioned technical problems, embodiments of this application provide a method for displaying whiteboard writing. This method incorporates speech recognition results as semantic supplementation and combines them with handwriting recognition results to achieve structuring of the whiteboard content. This improves the readability, structure, and semantic integrity of the whiteboard content, facilitating user understanding and reuse of the content after the meeting. Furthermore, the structured whiteboard content retains the original handwriting style, satisfying both the need for logical clarity and preserving the user's personalized expression, enhancing user emotional identification and willingness to use the content, and improving the user experience.

[0038] The whiteboard display method provided in this application is applied to an electronic device. This electronic device can be an interactive flat panel, a conference terminal, or other interactive device; this embodiment does not limit the application to this type. For ease of explanation, the electronic device will be described below using an interactive flat panel as an example.

[0039] Figure 1 This is a schematic diagram of the display interface of an interactive flat panel provided in an embodiment of this application.

[0040] For example, such as Figure 1 As shown, the interactive whiteboard displays slide page 101 and sidebar 102. Figure 1The display interface of the interactive tablet indicates that the interactive tablet has not opened the whiteboard application interface. The side bar 102 includes a first button 1021 for opening the whiteboard application interface. When the user wants to open the whiteboard application interface, the first button 1021 can be clicked. The whiteboard application software is installed in the interactive tablet, and the whiteboard application software is used to provide the whiteboard application interface. The whiteboard application software can be pre-installed in the interactive tablet, or can be a writing application software downloaded and installed from a third-party device or a server when the interactive tablet is started. The third-party device is not limited in the embodiment.

[0041] Figure 2 FIG. 1 is a schematic diagram of a whiteboard application interface provided by an embodiment of the present application.

[0042] For example, as described above, when it is detected that the user clicks the first button 1021, the display interface of the interactive tablet can display the whiteboard application interface 201. The whiteboard application interface 201 includes a tool bar 202, and the tool bar 202 includes a quit button 2021 for triggering the whiteboard application interface 201 to quit, a writing button 2023 for triggering a writing operation on the whiteboard application interface 201, an erasing button 2024 for triggering an erasing operation on the writing traces on the whiteboard application interface 201, an inserting button 2025 for triggering an inserting operation (such as inserting an image, a table, etc.) on the whiteboard application interface 201, a save button 2022 for triggering a saving operation on the board content of the whiteboard application interface 201, and the like. In a specific implementation, other function buttons can also be arranged in the tool bar 202, and the present embodiment does not limit this.

[0043] Considering that there can be board content in the conference scenario and the teacher teaching scenario, the display method of the board content in the present embodiment can be applied to the conference scenario, so as to facilitate the understanding and reuse of the board content by the participants after the conference ends, and can also be applied to the teacher teaching scenario, so as to facilitate the understanding and reuse of the board content by the students or the teacher after the classroom teaching ends.

[0044] Figure 3 FIG. 5 is a schematic flowchart of a display method of a board content provided by an embodiment of the present application.

[0045] For example, as shown in FIG. 5, the display method of the board content includes the following steps 301 to 305: Figure 3

[0046] Step 301: Display a whiteboard application interface.

[0047] Step 302: In response to a recognition instruction on the whiteboard application interface, recognize original writing traces on the whiteboard application interface to obtain a writing trace recognition result.

[0048] ​Step 303: Obtain the voice information recorded in the writing process, and identify the voice information to obtain a voice recognition result.

[0049] Step 304: Generate structured board content according to the handwriting recognition result and the voice recognition result; wherein the structured board content has the same writing style as the original writing handwriting.

[0050] Step 305: Display the structured board content on the whiteboard application interface.

[0051] In Figure 3 In the embodiment shown, the input information of multiple dimensions (original writing handwriting and voice information) is fused to realize semantic understanding and structured reconstruction of the board content, rather than just staying at the visual level of beautification. Specifically, by recognizing the original writing handwriting, the voice information of the user in the writing process is recognized at the same time as a context supplement, which is conducive to improving the accuracy of recognition and reducing the burden of manual correction caused by OCR recognition errors. By combining the voice recognition result, it is beneficial to supplement the semantic of the handwriting recognition result, and to fuse and associate the handwriting recognition result and the voice recognition result semantically, so as to construct a structured board content that is clear and logical, and to make up for the semantic loss problem caused by relying on simple text alone, so that the conference context can be accurately restored after the conference, and subsequent review and discussion are facilitated. Moreover, the structured board content retains the writing style of the original writing handwriting, which not only meets the demand for clear logic, but also retains the user's personalized expression, enhancing the user's emotional recognition and use willingness.

[0052] The specific implementation of each step in the embodiment shown in Figure 3 will be described below:

[0053] In step 301, the interactive tablet displays the whiteboard application interface on its display area after detecting an opening instruction of the whiteboard application interface. As shown in Figure 1 When the interactive tablet detects that the user clicks the first button 1021, it displays the whiteboard application interface, so that the display interface of the interactive tablet is switched to the whiteboard application interface as shown in Figure 2 .

[0054] In step 302, the interactive tablet starts to recognize the original writing handwriting on the whiteboard application interface to obtain a handwriting recognition result when it detects a recognition instruction of the whiteboard application interface. The triggering mode of the above-mentioned recognition instruction includes but is not limited to clicking a button, tapping a screen, pen interaction, etc. In a conference scenario, the user may trigger the recognition instruction after the conference writing process is completed.

[0055] In one possible implementation, refer to Figure 2The toolbar 202 can further include an identification button (not shown in the figure). If it is detected that the user clicks the identification button, it is determined that the identification instruction is detected, thereby automatically triggering the handwriting recognition. The triggering manner is intuitive and convenient for user operation.

[0056] In another possible implementation, if it is detected that the user draws a specific gesture (such as “circle selection + sliding”, “circle drawing”, “tapping”, etc.) in the writing area or the specified area, it is determined that the identification instruction is detected, thereby automatically triggering the handwriting recognition. The triggering manner does not need to leave the writing area for operation, and the operation efficiency is improved.

[0057] In another possible implementation, in a case where the user uses a writing pen to write in the whiteboard application interface, if it is detected that the user presses a specific key on the writing pen, it is determined that the identification instruction is detected, thereby automatically triggering the handwriting recognition. This is conducive to quickly triggering the handwriting recognition.

[0058] In a specific implementation, the identification instruction can also be implemented by a voice instruction, a remote control, and a gesture camera recognition, and the like. The handwriting recognition is triggered by multiple interaction manners, the flexibility of user operation and the recognition efficiency are improved, and the handwriting recognition is suitable for different use scenarios and user groups.

[0059] The original handwriting refers to a continuous or discontinuous handwriting track formed by the user writing on the whiteboard application interface by using a finger or a writing pen. The original handwriting reflects the writing intention of the user. It can be understood that the user can express the writing intention by various symbols and characters when writing. Therefore, the original handwriting can include original symbols and original characters.

[0060] The original symbol refers to a graphical element drawn by the user in the writing process to express a logical relationship, a structural relationship, or an indication intention. The original symbol usually does not directly represent a language meaning, but has an auxiliary understanding or information organization function. The original symbol can be recognized as a structured graphical element, used to build information organization structure, flowchart, mind map, and the like, and is an important part of semantic understanding and layout reconstruction in the whiteboard application.

[0061] The original character refers to a character or character combination drawn by the user in the writing process by using a finger or a writing pen to express a language meaning. The original character can include Chinese characters, English characters, and characters of other languages, and the like. The original character is a main carrier for the user to express core content. Subsequently, the original character can be converted into structured text by a text recognition algorithm such as OCR, and is used for information extraction, content retrieval, document generation, and the like.

[0062] The handwriting recognition result includes semantic information of the original handwriting, representing the actual meaning expressed by the handwriting content. Both the original symbol and the original character have their respective semantic information.

[0063] For the recognition of the original text, the original text can be recognized by using OCR or a deep learning model (such as a CNN, an RNN, or a Transformer), and semantic information of the original text can be obtained by performing semantic understanding on the recognized text.

[0064] For the recognition of the original symbol, the original symbol can be recognized by using image recognition or a machine learning model, a symbol style of the original symbol can be obtained, and semantic information of the original symbol can be determined by querying a corresponding relationship table of the symbol style and the semantic information according to the symbol style.

[0065] For example, the corresponding relationship table of the symbol style and the semantic information can refer to Table 1 shown in the following table:

[0066] Table 1

[0067]

[0068]

[0069]

[0070] For example, in the process of recognizing the original writing trace on the whiteboard application interface, an indication window representing the screen recognition can be displayed on the whiteboard application interface, so that the user can intuitively observe the state of being currently in the screen recognition. The following describes the whiteboard application interface in the process of recognizing the original writing trace in combination with Figure 4 and Figure 5 .

[0071] Figure 4 is a schematic diagram of a whiteboard application interface before recognizing the original writing trace.

[0072] For example, as shown in Figure 4 , the whiteboard application interface includes an original writing trace 401 and a toolbar 202. The original writing trace 401 includes original text and original symbols. The original text includes learning machine, home screen, school, public, personal, and outdoor, and the original symbols include single-sided parentheses, arrow, triangle, circle, square, cross, checkmark, and straight line. The toolbar 202 includes a plurality of function buttons, such as a quit button 2021, a save button 2022, a writing button 2023, an erasing button 2024, and an insertion button 2025.

[0073] Figure 5 is a schematic diagram of a whiteboard application interface in the process of recognizing the original writing trace.

[0074] For example, as shown in Figure 5As shown, the whiteboard application interface includes an instruction window 501, original writing traces 401, and a toolbar 202, and the instruction window 501 displays prompt information "screen recognition in progress". When a recognition instruction for the whiteboard application interface is detected, a visual feedback layer can be displayed on the whiteboard application interface. The visual feedback layer can be understood as a semi-transparent layer overlaid on the whiteboard application interface. The visual feedback layer displays the instruction window 501, which displays the prompt information "screen recognition in progress", and the original writing traces 401 can be observed through the visual feedback layer. The user can still see the original writing traces 401 through the visual feedback layer, avoiding the sense of information loss. This approach is helpful to guide the user's attention to the recognition state and prevent the user from misoperation during the recognition process.

[0075] In some embodiments, the original writing traces include original symbols and original text; and the recognition of the original writing traces on the whiteboard application interface to obtain the writing recognition result includes: recognizing the original symbols written by the user on the whiteboard application interface to obtain the writing recognition result of the original symbols; and recognizing the original text within a preset range centered on the recognized original symbols to obtain the writing recognition result of the original text.

[0076] The preset range can be a surrounding area centered on the original symbol, which can be a circular area or a rectangular area. For example, the preset range is a circular area centered on the original symbol and having a preset distance as a radius, and the preset distance can be a pixel distance, such as 30-80 px. For another example, the preset range is a fixed-size rectangular frame surrounding the original symbol, which is suitable for structured arrangement content recognition. Optionally, the size of the preset range can also be adjusted according to the writing density and user writing habits, such as: a small range (30-50 px) is suitable for compact writing (such as formulas, flowcharts), and a medium range (60-80 px) is suitable for relatively loose writing (such as meeting notes, mind maps).

[0077] For example, for a bracket type original symbol (such as (), [], and {}), the original text within the brackets is recognized centered on the recognized bracket type original symbol, and is parsed from inside to outside according to the nesting relationship to construct a structured expression, for example: (text1+text2); wherein text1+text2 can be understood as the original text within the preset range centered on the original symbol "()" recognized.

[0078] For example, for an arrow type original symbol (such as →, ), the original text before and after the arrow is recognized centered on the recognized arrow type original symbol, and a "cause→effect" logical chain is constructed, for example: "demand analysis→scheme design"; wherein "demand analysis" and "scheme design" can be understood as the original text within the preset range centered on the original symbol "→" recognized.

[0079] The handwriting recognition result of the original symbol includes a symbol category, a symbol name, a symbol style, and semantic information of the original symbol. The handwriting recognition result of the original text includes semantic information of the original text, a coordinate position of the original text in the whiteboard application interface, writing style information, confidence information, context association information, language information, and the like. The context association information refers to an association relationship between the original text and original symbols around the original text, such as the original text being located in front of an arrow or the original text being nested in brackets. The language information refers to a language category to which the original text belongs, such as Chinese, English, or a Chinese-English mixture. The semantic information of the original text includes content of the original text and semantic meanings thereof that are recognized.

[0080] In the above embodiment, the original symbol is first recognized, and then the original text around the original symbol is recognized. The original symbol itself has a structural semantic, which can guide the recognition of the content of the text around the original symbol and form a logical association. The handwriting content of a user is usually irregularly arranged. By using the original symbol for guidance, the intention of the user can be more accurately understood, and the accuracy of the handwriting recognition result can be improved.

[0081] In step 303, it can be understood that a user usually also explains and describes the content written by the user in the process of performing a writing operation on the whiteboard application interface. Based on this, while the original writing handwriting on the whiteboard application interface is recognized, voice information recorded by an interactive tablet in the writing process of the user can also be obtained, and the voice information can be recognized to obtain a voice recognition result. The voice information recorded by the interactive tablet can be understood as voice information recorded by the interactive tablet in the writing process of the user. The voice information can be understood as a continuous audio stream. By way of example, the voice content of the recorded voice information is recognized by a voice-to-text technology, the voice content is converted into text content, and the voice recognition result is obtained according to the converted text content. The voice recognition result can be used to supplement and correct the handwriting recognition result, and the handwriting recognition result and the voice recognition result can be combined to construct complete semantic expression.

[0082] In step 304, the interactive tablet combines the handwriting recognition result and the voice recognition result to generate structured board content. The structured board content has the same writing style as the original writing handwriting. The structured board content can be understood as board content that has semantic association and clear structure and is generated by fusing the handwriting recognition result and the voice recognition result on the premise that the original writing style (such as font, layout, color, stroke thickness, and the like) is retained.

[0083] Exemplarily, according to the handwriting recognition result and the speech recognition result, the content of the board writing can be automatically organized into structured board writing content with hierarchical structure and logical relationship through semantic analysis and structure modeling, and the structured board writing content can be in the form of a mind map.

[0084] Exemplarily, in order to make the structured board writing content have the same writing style as the original writing handwriting, a style transfer model can be used so that the writing style of the original writing handwriting can be preserved, or a vector graphics rendering technology can be directly used to draw the structured board writing content according to the writing style of the original writing handwriting.

[0085] In some embodiments, the implementation of step 304 includes steps 3041 to 3042 as follows:

[0086] Step 3041: correcting the original writing handwriting according to the speech recognition result and the handwriting recognition result to obtain the target writing handwriting after correction.

[0087] Considering that there can be writing errors (such as misspelling), writing simplification (such as omission of part of the content), and unattractive writing during the writing process, some redundant information or draft traces will inevitably be generated during the writing process, affecting the neatness and readability of the final board writing. Based on this, the original writing handwriting is corrected according to the speech recognition result and the handwriting recognition result to obtain the target writing handwriting after correction. The target writing handwriting after correction has a certain degree of improvement in terms of integrity, accuracy, aesthetics, and logic compared to the original writing handwriting.

[0088] Exemplarily, the speech recognition result includes semantic information of a plurality of speech segments, and the handwriting recognition result includes semantic information of a plurality of original writing handwritings; the implementation of the above step 3041 includes S11 to S13 as follows:

[0089] S11: determining the time stamp of the plurality of original writing handwritings and determining the time stamp of the plurality of speech segments.

[0090] Exemplarily, the time sequence of the user's writing action can be recorded through the interactive tablet, and a time stamp is assigned to each original writing handwriting to represent the writing start time and the writing end time of the original writing handwriting.

[0091] Exemplarily, considering that an original writing handwriting has a certain continuity in time, an original writing handwriting can be understood as: the original writing handwriting written by the user within a continuous period of time, which can include a plurality of original characters and / or a plurality of original symbols. For example, the user writes “f(x)=x 2 ” within 3 seconds to form a continuous original writing handwriting.

[0092] For example, a piece of original writing handwriting can be a formula (such as "f(x) = x 2 +y 2 "), a logical expression (such as "A→B"), a keyword or phrase (such as "requirement analysis", "solution").

[0093] For example, the recorded voice information can be divided into multiple voice segments, and the timestamps of the multiple voice segments are determined to represent the voice start time and the voice end time of the voice segment. That is, the recorded voice information is segmented and recognized, each voice segment is assigned a timestamp, and the voice recognition result of each voice segment is obtained.

[0094] Optionally, the continuous voice information can be divided into multiple voice segments according to the time information of the writing operation. The time information of the writing operation can include the writing start time and the writing duration of each piece of original writing handwriting. The writing duration is the duration between the writing start time and the writing end time.

[0095] For example, the writing start time is taken as the voice start time of the voice segment, the writing duration is taken as the length of the voice segment, and the voice segment corresponding to the time period is extracted from the voice information. For example, the user starts writing "f(x) = x 2 +y 2 " at 15:30:02, and the writing duration is 5 seconds. Therefore, the voice information from 15:30:02 to 15:30:07 is extracted from the voice information as a voice segment.

[0096] For example, the target time point before the writing start time is taken as the voice start time of the voice segment, the target time length greater than the writing duration is taken as the length of the voice segment, and the voice segment corresponding to the time period is extracted from the voice information. For example, the user starts writing "f(x) = x 2 +y 2 " at 15:30:02, and the writing duration is 5 seconds. Therefore, the voice information from 15:28:00 to 15:31:00 is extracted from the voice information as a voice segment.

[0097] S12: According to the timestamps of the multiple pieces of original writing handwriting and the timestamps of the multiple voice segments, a mapping relationship between the multiple voice segments and the multiple pieces of original writing handwriting is established.

[0098] According to the timestamps, the voice segment is time-aligned with the original writing trace to establish a mapping relationship. The mapping strategy can be a time overlap matching strategy or a time proximity matching strategy.

[0099] The time overlap matching refers to that when the timestamps of the voice segment and the timestamps of the original writing trace overlap, the two are considered to be related, and then a mapping relationship between the voice segment and the original writing trace is established.

[0100] The overlap of the timestamps of the voice segment and the timestamps of the original writing trace can be understood as that the time period between the start time and the end time of the voice segment overlaps with the time period between the writing start time and the writing end time of the original writing trace.

[0101] The time proximity matching refers to that when the time interval between the timestamps of the voice segment and the timestamps of the original writing trace is less than a preset threshold (such as 1 second), the two are considered to be related, and then a mapping relationship between the voice segment and the original writing trace is established. The time interval between the timestamps of the voice segment and the timestamps of the original writing trace being less than the preset threshold can be understood as that the time interval between the voice start time of the voice segment and the writing start time of the original writing trace is less than the preset threshold, and / or the time interval between the voice end time of the voice segment and the writing end time of the original writing trace is less than the preset threshold.

[0102] S13: For the voice segment and the original writing trace having the mapping relationship, the original writing trace is corrected according to the semantic information of the voice segment and the semantic information of the original writing trace to obtain a corrected target writing trace.

[0103] It can be understood that the voice segment and the original writing trace having the mapping relationship usually have certain semantic relevance, and therefore, for the voice segment and the original writing trace having the mapping relationship, the semantic information of the voice segment can be used to more accurately correct the original writing trace having the mapping relationship with the voice segment, thereby improving the accuracy of the corrected target writing trace.

[0104] For example, according to the semantic information of the voice segment and the semantic information of the original writing trace, one or any combination of the following steps is performed to correct the original writing trace: removing redundant writing traces in the original writing trace, correcting erroneous writing traces in the original writing trace, supplementing omitted writing traces, and beautifying writing traces to be beautified in the original writing trace; wherein the writing style of the beautified writing trace is the same as that of the un-beautified writing trace.

[0105] By comparing the semantic information of the voice segment with the semantic information of the original writing trace, redundancy, error, omission or non-standard content in the writing content is identified, and the original writing trace is corrected in at least one dimension according to the above. The at least one dimension correction includes one or any combination of the following: redundancy trace removal, error trace correction, omission content supplement, and writing trace beautification. The following will introduce these correction methods in detail:

[0106] In some embodiments, the removal of the redundant writing trace in the original writing trace can be understood as follows: if the semantic information of the voice segment represents that there is a predetermined redundant keyword in the voice segment, the redundant content indicated by the redundant keyword is determined based on the context information of the redundant keyword, the redundant writing trace in the original writing trace is determined based on the redundant content and the semantic information of the original writing trace, and the redundant writing trace is removed. The redundant writing trace may be redundant original text and / or original symbols, and the time difference between the time point corresponding to the redundant writing trace and the time point at which the redundant keyword is identified in the voice segment is less than a predetermined time difference, so as to ensure that the redundant writing trace corresponding to the redundant keyword in the voice segment is accurately removed in the original writing trace.

[0107] Wherein, the redundant keyword can be "draft", "delete", "randomly drawn", etc.

[0108] For example, there is semantic information "this arrow is randomly drawn" in the voice segment, then based on the context information of the redundant keyword "randomly drawn", the redundant content indicated by the redundant keyword is determined as "arrow", and then based on the semantic information of the original writing trace, the redundant writing trace in the original writing trace is determined as "arrow", so as to remove "arrow" in the original writing trace. The time difference between the time point corresponding to the redundant writing trace "arrow" and the time point at which the redundant keyword "randomly drawn" is identified in the voice segment is less than a predetermined time difference, so as to ensure that the redundant writing trace "arrow" corresponding to "randomly drawn" in the voice segment is accurately removed in the original writing trace. For example, there is semantic information "the formula calculation part is a draft, not to be seen first" in the voice segment, then based on the context information of the redundant keyword "draft", the redundant content indicated by the redundant keyword is determined as "formula calculation part", and then based on the semantic information of the original writing trace, the redundant writing trace in the original writing trace is determined as "formula calculation part", so as to remove "formula calculation part" in the original writing trace.

[0109] In some embodiments, the individual points and irregular chaotic lines in the original writing trace can also be located by image recognition, and these individual points and irregular chaotic lines are removed as redundant writing traces.

[0110] In some embodiments, correcting the erroneous writing in the original writing can be understood as: determining the erroneous writing in the original writing by comparing the semantic information of the voice segment and the semantic information of the original writing, and correcting the erroneous writing to the correct writing. The erroneous writing can be an erroneous original character, i.e., a wrong character, or an erroneous original symbol. For example, the semantic information of the original writing includes: the character content of the original writing is “ate a bolo at noon”, and the semantic information of the voice segment represents that the current discussion is about food. Based on this, it can be determined that there is a wrong character “bolo” in the original writing, and “bolo” in the original writing is corrected to “bo”.

[0111] In some embodiments, supplementing the omitted writing can be understood as: determining the missing content in the original writing by comparing the semantic information of the voice segment and the semantic information of the original writing, and supplementing the missing content in the original writing. The missing content is the omitted writing. The omitted writing can be a character or a symbol. The following will be described by taking the omitted writing as a character as an example: for example, the semantic information of the voice segment includes: “focus on market demand forecast”, and the semantic information of the original writing omits “market demand forecast” and only retains “focus on”. Based on this, it can be determined that the omitted writing is “market demand forecast”, and “market demand forecast” is supplemented after “focus on” in the original writing.

[0112] In some embodiments, beautifying the writing to be beautified in the original writing can be understood as: determining the writing to be beautified in the original writing by comparing the semantic information of the voice segment and the semantic information of the original writing, and beautifying the writing to be beautified by using an image processing algorithm or a style transfer model. The writing to be beautified retains the same writing style (such as font, stroke thickness, inclination angle, color, etc.) as the writing to be beautified after beautification. The writing to be beautified can be original characters with unclear connected writing, messy strokes, and messy handwriting. For example, if the semantic information of the voice segment includes “very happy”, the semantic information of the original writing also includes “very happy”, but the writing of “happy” in the original writing is messy, then the writing to be beautified in the original writing is determined to be “happy” the two original characters, and the writing to be beautified is “happy” the two original characters to make the writing of “happy” the two original characters neat and not messy after beautification.

[0113] In the above embodiments, when correcting the original writing trace based on the mapping relationship, if the semantic information between the voice segment and the original writing trace is not the same, the error writing trace in the original writing trace is corrected according to the semantic information of the voice segment. If the semantic information of the voice segment is complete, and the semantic information of the original writing trace is missing, the missing writing trace is supplemented into the original writing trace. On the premise that the semantic information between the voice segment and the original writing trace is consistent, the writing trace to be beautified in the original writing trace is stroke, and the non-standard expression in the original writing trace is replaced by a more standard expression. If the semantic information of the voice segment clearly indicates that a certain writing trace is "draft", "invalid", "randomly drawn", the writing content of this part is removed in combination with the writing recognition result of the original writing trace.

[0114] By adopting the correction mode of redundant writing trace removal, unnecessary or repetitive parts in the original writing trace, such as redundant lines, repeatedly written words, etc., can be identified and removed, which helps to improve the clarity and readability of the target writing trace after correction. By adopting the correction mode of error writing trace correction, the error writing trace (such as spelling error, wrong character, etc.) in the original writing trace can be automatically detected and corrected, which improves the accuracy of the target writing trace after correction and avoids misleading the user by the error writing trace. By adopting the correction mode of supplementing omitted writing trace, the omitted content in the writing process can be automatically identified and supplemented, which is beneficial to improve the semantic integrity and logical coherence of the target writing trace after correction. By adopting the correction mode of writing trace beautification, the original writing trace can be adjusted to be more beautiful and standard, while retaining the original writing style, which meets the personalized needs of the user.

[0115] Step 3042: generating structured blackboard content according to the target writing trace.

[0116] For example, the target writing trace can be structured to generate structured blackboard content. Structured processing refers to a way of organizing and presenting blackboard content in a logical and clear manner. The structured blackboard content obtained by structured processing is more clear in structure, more distinct in logical structure, and higher in readability.

[0117] Exemplarily, the target writing handwriting is structurally processed, including: according to semantic information of the target symbol and semantic information of the target character around the target symbol, establishing a structural connection between the target symbol and the target character. The structural connection refers to organizing and connecting the original symbol and the original character according to their semantic relationship and logical structure on the basis of understanding semantics, so as to form a structured information network with clear hierarchy, logical clarity and strong readability. Then, taking the writing style of the original writing handwriting as a mimic object, the target symbol and the target character connected are processed to generate a structured board content, so that the structured board content and the writing style of the original writing handwriting are the same.

[0118] In some embodiments, considering that the original writing handwriting includes original characters and original symbols, and the target writing handwriting is obtained by modifying the original writing handwriting, the modification includes the above-mentioned redundant handwriting removal, error handwriting correction, omitted content supplement and writing handwriting beautification, therefore, the target writing handwriting can include modified target characters and modified target symbols, and the target symbol is surrounded by target characters associated with it in semantics. Taking the writing style of the original writing handwriting as a mimic object, the target symbol and the target character are processed to generate a structured board content.

[0119] In step 305, after generating the structured board content, the structured board content can be displayed on the whiteboard application interface. The structured board content is more clear in structure, and is convenient for viewing and understanding key information.

[0120] In some embodiments, the above-mentioned displaying the structured board content on the whiteboard application interface includes: in the case of reserving the original writing handwriting on the whiteboard application interface, displaying the structured board content on the whiteboard application interface in a first display mode; wherein the first display mode is different from the display mode of the original writing handwriting on the whiteboard application interface.

[0121] Wherein, the original writing handwriting can also be understood as the original board content. On the basis of reserving the original board content, the structured board content is displayed in the first display mode, realizing the differentiated display of the original board content and the structured board content, and facilitating the user to intuitively see which content belongs to the original board content and which content belongs to the structured board content.

[0122] For example, the first display method is highlighting. Highlighting refers to a display method that visually distinguishes the structured blackboard content from the original handwriting by changing its visual representation (such as color, transparency, background, etc.). In other words, the structured blackboard content is displayed using highlighting, while the original handwriting can be displayed without highlighting. The structured blackboard content includes the aforementioned target symbols and target text; therefore, displaying the structured blackboard content according to the first display method can also be understood as highlighting the target symbols and target text.

[0123] By highlighting the structured whiteboard content, users can easily identify which parts are "original handwriting" and which are "structured whiteboard content," making it simpler and more convenient to visually separate "original handwriting" from "structured whiteboard content."

[0124] For example, the original handwriting can also be displayed as an optional item, allowing users to choose whether or not to display it. For instance, the whiteboard application interface can display a selection button, allowing users to choose whether to retain the original handwriting display to keep the whiteboard application interface clean or focus on key content.

[0125] Figure 6 This is a schematic diagram of a structured whiteboard content display interface provided in an embodiment of this application.

[0126] Examples such as Figure 6 As shown, the whiteboard application interface includes original whiteboard content 601 and structured whiteboard content 602. Compared to the original whiteboard content 601, the structured whiteboard content 602 is highlighted on the whiteboard application interface, making the whiteboard structure clearer and easier to view and understand key information.

[0127] Figure 7 This is a schematic flowchart illustrating another method for displaying whiteboard text provided in an embodiment of this application.

[0128] For example, such as Figure 7 As shown, the method for displaying the blackboard writing includes the following steps 701 to 711:

[0129] Step 701: The system detects that the user has opened the whiteboard application software and displays the whiteboard application interface.

[0130] Step 702: Received recognition command for whiteboard application interface.

[0131] Step 703: Activate the trajectory layout analysis algorithm to classify the original writing trajectory.

[0132] For example, the classification results of the original writing strokes include original symbols and original characters, and the original symbols further include arrows, parentheses, special graphic symbols, alphanumeric characters, dotted lines, professional symbols, etc. The classification results of the original symbols and the semantic information of the symbols can also be referred to Table 1, and details are not repeated here.

[0133] Step 704: Automatically searching the trajectory signals of each original writing stroke and marking a time stamp.

[0134] The trajectory signal refers to a series of data recorded by the interactive panel about the writing action when the user writes or draws on the whiteboard application interface. These data usually include the coordinate position, pressure value, angle and other possible related information at each time point in the writing process, and each coordinate position is associated with a time stamp for recording the exact time when the position is recorded. Based on this, a time stamp can be marked for each original writing stroke to indicate the writing start time and the writing end time of the original writing stroke.

[0135] Step 705: Performing semantic analysis on the original writing strokes to obtain semantic information of the original writing strokes.

[0136] Step 706: Segmenting the recorded voice information and marking a time stamp for each voice segment.

[0137] Step 707: Using voice recognition technology to perform segmented semantic analysis on the voice information to obtain semantic information of each voice segment.

[0138] Step 708: Aligning the voice segments and the original writing strokes based on the time stamp to establish a mapping relationship between the voice and the writing.

[0139] Step 709: Based on the mapping relationship, correcting the original writing strokes to obtain the corrected target writing strokes.

[0140] By establishing a mapping relationship between the original writing strokes and the voice segments, it is beneficial to enhance the understanding of the content of the original writing by combining the voice information.

[0141] Step 710: Simulating the writing style of the writing strokes by the writing imitation technology based on the corrected target writing strokes to obtain the structured board content.

[0142] Step 711: On the basis of retaining the original writing strokes, performing highlight processing and presentation on the structured board content.

[0143] By imitating the writing style of the original writing strokes through the writing imitation technology, the generated key writing, i.e., the structured board content, is highlighted and presented on the basis of retaining the original writing strokes, so that the board structure is clear and the key information is easy to view and understand.

[0144] In some embodiments, the structured board content includes a plurality of theme labels each corresponding to theme content; and displaying the structured board content on the whiteboard application interface includes: displaying the plurality of theme labels on the whiteboard application interface; in response to a click operation on a target theme label, displaying theme content corresponding to the target theme label on the whiteboard application interface in a second display mode, and displaying theme content corresponding to non-target theme labels on the whiteboard application interface in a third display mode; wherein the target theme label is any one of the plurality of theme labels.

[0145] The theme label is a summary identifier obtained after the structured board content is subject to theme classification, and is usually a short keyword or phrase. The theme content corresponding to the theme label refers to specific explanations, descriptions, data or logic, etc. under the theme label, which is an expanded explanation of the theme label, and different theme contents can be independent blocks or associated with each other. In specific implementation, the theme label can serve as an index of the theme content, and the theme content is detailed content under the index.

[0146] It can be understood that the structured board content can involve theme content of different theme labels, and different theme labels can be understood as different categories. The theme content corresponding to different theme labels can also be understood as theme content of different categories. The structured board content is different, and the plurality of theme labels corresponding thereto are also different. For example, the plurality of theme labels can include labels representing categories such as theme, scheme, background, speaker, and handwriting color. The theme content corresponding to the theme label can be understood as theme content belonging to the theme label.

[0147] In order not to affect the normal display of the board content, the plurality of theme labels can be displayed in the edge area of the whiteboard application interface, such as the top area or the bottom area of the whiteboard application interface. Each theme label is clickable, and the style (such as color, icon, font size) of the theme label can be pre-set. If a click operation of a user on a theme label is detected, it indicates that the user expects to focus on viewing the theme content corresponding to the theme label, and the theme label is taken as a target theme label, the theme content corresponding to the target theme label is displayed on the whiteboard application interface in a second display mode, and the theme content corresponding to non-target theme labels is displayed on the whiteboard application interface in a third display mode. The target theme label refers to the theme label clicked by the user, and the non-target theme label refers to other theme labels in the plurality of theme labels except the target theme label, i.e. the theme label not clicked by the user.

[0148] The above embodiments, by displaying multiple topic tags on the whiteboard application interface, allow users to quickly view content on topics of interest by clicking on a specific tag, enhancing interactivity. By clicking on different topic tags, users can easily navigate back and clearly view the structured presentation of the content under each tag, effectively achieving efficient information retrieval and understanding, and meeting diverse needs for accessing whiteboard content.

[0149] For example, the second display method is a highlighted display, and the third display method is a non-highlighted display. That is, the topic content corresponding to the topic tag selected by the user will be highlighted on the whiteboard application interface, and the topic content corresponding to the topic tag not selected by the user will be non-highlighted on the whiteboard application interface.

[0150] Figure 8 This is a schematic diagram of a whiteboard application interface displaying theme tags, provided in an embodiment of this application.

[0151] For example, such as Figure 8 As shown, the bottom area of ​​the whiteboard application interface displays four topic tabs: Baby Stroller Visual Inspection Solution, Technology Implementation Path, Market Demand Forecast, and User Scenarios and Pain Points. Figure 8 In the example, the user clicked on the topic tag "baby stroller visual inspection solution". Therefore, the highlighted topic content is topic content 801 corresponding to the topic tag "baby stroller visual inspection solution". The non-highlighted topic content includes all other topic content except for topic content 801 corresponding to the topic tag "baby stroller visual inspection solution".

[0152] Figure 9 This is a schematic diagram of another whiteboard application interface displaying topic tags provided in an embodiment of this application.

[0153] For example, such as Figure 9 As shown, the bottom area of ​​the whiteboard application interface displays four topic tabs and... Figure 8 The four hashtags in the text are the same. Figure 9 In the example, the user clicked on the topic tag "Technical Implementation Path". Therefore, the highlighted topic content is topic content 901 corresponding to the topic tag "Technical Implementation Path". The non-highlighted topic content includes all other topic content except topic content 901 corresponding to the topic tag "Technical Implementation Path".

[0154] For example, before displaying multiple topic tags on the whiteboard application interface, the method further includes: classifying the structured whiteboard content into topics based on handwriting recognition results and speech recognition results to obtain multiple topic tags.

[0155] In this example, the handwriting recognition result and the speech recognition result are combined to extract a theme, obtain a plurality of theme tags, and obtain theme content corresponding to each of the plurality of theme tags. For example, a classification model can be trained using a labeled data set, which can automatically learn and identify which topic different types of information belong to. The handwriting recognition result and the speech recognition result are input into the classification model, and the classification model can output a plurality of theme tags.

[0156] For example, the original writing handwriting includes original symbols, the handwriting recognition result includes semantic information of the original symbols, region features of the original writing handwriting distribution, and color information of the original writing handwriting. The theme classification of the structured board content based on the handwriting recognition result and the speech recognition result to obtain a plurality of theme tags includes: extracting keywords based on the speech recognition result; and classifying the structured board content based on the semantic information of the original symbols, the region features, the color information, and the keywords to obtain a plurality of theme tags.

[0157] The writing area of the whiteboard application interface can be large, and the user can stay in a certain specific area during writing. Therefore, the handwriting recognition result can include region features of the original writing handwriting distribution. The region features can be understood as the distribution density of the original writing handwriting in the writing area of the whiteboard application interface. The area with high distribution density is marked as a dense writing area, and the writing content such as original text and original symbols in the dense writing area is relatively concentrated. The writing content in the dense writing area often represents the user's explanation focus or core logic, and the writing content in the dense writing area is given priority in theme classification, which is beneficial to improve the classification accuracy.

[0158] The user can choose different colors for writing during writing. Therefore, the handwriting recognition result can also include color information of the original writing handwriting, which can include black, red, green, yellow, and the like.

[0159] For example, the speech recognition result can include speech text obtained by transcribing speech information, and high-frequency and summarized keywords can be extracted from the speech text using speech recognition technology. The extracted keywords can be used for subsequent theme classification. The keywords can reflect the user's explanation focus, and can be combined with the handwriting recognition result to assist theme classification.

[0160] After obtaining the semantic information of the original symbols, the regional features of the original writing track distribution, the color information, and the keywords extracted from the speech recognition result, the structured board writing content is classified according to the semantic information of the original symbols, the regional features, the color information, and the keywords, to obtain a plurality of theme labels. For example, a multi-modal classification model can be constructed, and the keywords, the semantic information of the original symbols, the regional features, and the color information are input into the multi-modal classification model, so as to output a plurality of theme labels through the multi-modal classification model.

[0161] In the above embodiment, the multi-modal information such as the semantic information of the original symbols, the regional features of the original writing track distribution, and the color information, and the keywords extracted from the speech recognition result is combined for theme classification, which is beneficial to improve the accuracy of theme classification.

[0162] Figure 10 is a schematic flowchart of another board writing display method provided by the embodiment of the present application.

[0163] For example, as shown in Figure 10 The board writing display method includes the following steps 1001 to 1013.

[0164] Step 1001: After the start of the meeting, it is detected that the user starts the meeting recording function.

[0165] Step 1002: The speech information collected during the meeting is transcribed in real time to obtain speech text.

[0166] Step 1003: Start the trajectory page analysis algorithm to classify the original writing track.

[0167] Step 1004: Mark the time stamp of the speech information and the original writing track.

[0168] Step 1005: Align the speech information and the original writing track based on the time stamp to establish a mapping relationship between the speech and the track.

[0169] Step 1006: Based on the mapping relationship, the original writing track is corrected to obtain the target writing track after correction.

[0170] Step 1007: The target writing track after correction is simulated to generate writing style track through track imitation technology to obtain structured board writing content.

[0171] Step 1008: Analyze the speech text and extract keywords through speech recognition technology.

[0172] Step 1009: Identify the original writing track to obtain the semantic information of the original symbols, the regional features of the distribution, and the color information.

[0173] Step 1010: According to the keywords, the semantic information of the original symbols, the area features of the dense writing area, and the color information, the structured board writing content is classified by theme, and a plurality of theme labels are generated and displayed.

[0174] Step 1011: It is detected that a user clicks a certain theme label.

[0175] Step 1012: The theme content corresponding to the theme label clicked by the user is highlighted.

[0176] Step 1013: The theme content corresponding to the theme label not clicked by the user is not highlighted.

[0177] In Figure 10 the embodiment shown, during the conference recording process, first, the speech information is transcribed in real time, and the recognition of the original symbols and the original text in the handwriting board writing (i.e., the original writing traces) is completed, and the accurate alignment of the handwriting board writing and the speech information is realized through the time stamp. Subsequently, the speech text is deeply analyzed by using the speech recognition technology, the high-frequency and summarized keywords are extracted, and the theme labels covering the theme, the scheme, the background, the speaker, the writing color and the like are generated by combining the symbol logic of the handwriting content (i.e., the semantic information of the original symbols), the area features of the dense writing area, and the color information. These theme labels are intuitively presented on the whiteboard application interface. When it is detected that a user clicks a certain theme label, based on the fine association of the time stamp and the theme content, the board writing area corresponding to the theme label is quickly located, and the theme content in the board writing area is highlighted and displayed with a distinctive visual effect. The user can conveniently backtrack and clearly view the structured presentation of the board writing content under different categories of theme labels, effectively realize the efficient retrieval and understanding of information, and meet the diversified conference recording review needs.

[0178] Figure 11 is a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0179] For example, as Figure 11 shown, the electronic device 1100 includes a memory 1101 and a processor 1102, wherein the memory 1101 stores executable program code, and the processor 1102 is configured to invoke and execute the executable program code to execute a display method of board writing.

[0180] In addition, an apparatus is also protected by an embodiment of the present application, which can include a memory and a processor, wherein the memory stores executable program code, and the processor is configured to invoke and execute the executable program code to execute a display method of board writing provided by an embodiment of the present application.

[0181] The embodiment can divide the functions of the device according to the method examples described above, for example, each function module can be divided, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware. It should be noted that the division of the modules in the embodiment is illustrative, and is only a logical function division. In actual implementation, another division mode can be used.

[0182] It should be understood that the device provided by the embodiment is used to execute the display method of the board described above, and thus the same effects as the implementation method described above can be achieved.

[0183] In the case of using an integrated unit, the device can include a processing module and a storage module. When the device is applied to an electronic device, the processing module can be used to control and manage the actions of the electronic device. The storage module can be used to support the electronic device to execute related program codes and the like.

[0184] The processing module can be a processor or a controller, which can implement or execute various exemplary logical blocks, modules and circuits shown in combination with the disclosure of the present application. The processor can also be a combination of computing functions, such as one or more microprocessor combinations, digital signal processing (DSP) and microprocessor combinations, and the like. The storage module can be a memory.

[0185] In addition, the device provided by the embodiment of the present application can be a chip, an assembly or a module. The chip can include a connected processor and a memory. The memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute the display method of the board provided by the above embodiment.

[0186] The embodiment also provides a computer readable storage medium, which stores computer program codes. When the computer program codes run on a computer, the computer executes the related method steps to implement the display method of the board provided by the above embodiment.

[0187] The embodiment also provides a computer program product, which makes the computer execute the related steps to implement the display method of the board provided by the above embodiment when the computer program product runs on the computer.

[0188] The device, computer readable storage medium, computer program product or chip provided by the embodiment are used to execute the corresponding method provided above, and thus the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be described here.

[0189] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0190] In the embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules or units is only a logical function division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0191] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A display method of board writing, characterized by, The method comprises: displaying a whiteboard application interface; in response to a recognition instruction for the whiteboard application interface, recognizing original writing traces on the whiteboard application interface to obtain a handwriting recognition result; obtaining voice information recorded in a writing process and recognizing the voice information to obtain a voice recognition result; generating structured board writing content according to the handwriting recognition result and the voice recognition result; wherein the structured board writing content is of the same writing style as the original writing traces; displaying the structured board writing content on the whiteboard application interface.

2. The method of claim 1, wherein, The displaying of the structured board writing content on the whiteboard application interface comprises: displaying the structured board writing content on the whiteboard application interface in a first display mode while retaining the original writing traces on the whiteboard application interface; wherein the first display mode is different from the display mode of the original writing traces on the whiteboard application interface.

3. The method of claim 2, wherein, The first display mode is high-light display.

4. The method of claim 1, wherein, The structured board writing content comprises theme content corresponding to each of a plurality of theme labels; The displaying of the structured board writing content on the whiteboard application interface comprises: displaying the plurality of theme labels on the whiteboard application interface; in response to a click operation on a target theme label, displaying theme content corresponding to the target theme label on the whiteboard application interface in a second display mode and displaying theme content corresponding to non-target theme labels on the whiteboard application interface in a third display mode; wherein the target theme label is any one of the plurality of theme labels.

5. The method of claim 4, wherein, The second display mode is high-light display and the third display mode is non-high-light display.

6. The method of claim 4, wherein, Before displaying the plurality of theme labels on the whiteboard application interface, the method further comprises: classifying the structured board writing content according to the handwriting recognition result and the voice recognition result to obtain the plurality of theme labels.

7. The method of claim 6, wherein, The original writing traces comprise original symbols, the handwriting recognition result comprises semantic information of the original symbols, region features of the original writing traces, and color information of the original writing traces; The classifying of the structured board writing content according to the handwriting recognition result and the voice recognition result to obtain the plurality of theme labels comprises: extracting keywords based on the voice recognition result; classifying the structured board writing content according to the semantic information of the original symbols, the region features, the color information, and the keywords to obtain the plurality of theme labels.

8. The method of claim 1, wherein, The original writing traces comprise original symbols and original text; The recognizing of the original writing traces on the whiteboard application interface to obtain a handwriting recognition result comprises: recognizing original symbols written by a user on the whiteboard application interface to obtain a handwriting recognition result of the original symbols; recognizing original text within a preset range centered on the recognized original symbols to obtain a handwriting recognition result of the original text.

9. The method of claim 1, wherein, The generating of structured board writing content according to the handwriting recognition result and the voice recognition result comprises: According to the voice recognition result and the handwriting recognition result, the original writing handwriting is corrected to obtain target writing handwriting after correction; According to the target writing handwriting, structured board writing content is generated.

10. The method of claim 9, wherein, The voice recognition result includes semantic information of multiple voice segments, and the handwriting recognition result includes semantic information of multiple original writing handwritings; The correction of the original writing handwriting according to the voice recognition result and the handwriting recognition result to obtain target writing handwriting after correction includes: Time stamps of the multiple original writing handwritings are determined, and time stamps of the multiple voice segments are determined; According to the time stamps of the multiple original writing handwritings and the time stamps of the multiple voice segments, a mapping relationship between the multiple voice segments and the multiple original writing handwritings is established; For voice segments and original writing handwritings with the mapping relationship, the original writing handwriting is corrected according to semantic information of the voice segment and semantic information of the original writing handwriting to obtain target writing handwriting after correction.

11. The method of claim 10, wherein, The correction of the original writing handwriting according to the semantic information of the voice segment and the semantic information of the original writing handwriting includes: According to the semantic information of the voice segment and the semantic information of the original writing handwriting, one or any combination of the following steps is performed to correct the original writing handwriting: Redundant handwriting existing in the original writing handwriting is removed; Error handwriting existing in the original writing handwriting is corrected; Writing handwriting to be beautified existing in the original writing handwriting is beautified; wherein the writing style of the writing handwriting after beautification is the same as that of the writing handwriting before beautification; Omitted writing handwriting is supplemented.

12. An electronic device, comprising: The electronic device includes: A memory for storing executable program code; A processor for calling and running the executable program code from the memory, so that the electronic device executes the method as claimed in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, when the computer program is executed, the method as claimed in any one of claims 1 to 11 is realized.

Images