Multimodal conference recording method, electronic device, storage medium, and program product

Abstract

This application discloses a multimodal meeting recording method, electronic device, storage medium, and program product. The method includes: synchronously acquiring handwritten data and meeting audio data, whereby the handwritten data includes handwriting trajectory data; identifying handwriting marks based on the handwriting trajectory data and obtaining the timestamps corresponding to the handwriting marks; aligning the meeting audio data with the handwriting marks and handwritten data to obtain aligned meeting data; and generating meeting records based on the handwriting marks, timestamps, and aligned meeting data. This application synchronously acquires handwritten data and meeting audio data, identifies handwriting marks through the handwriting trajectory data, aligns the meeting audio data with the handwriting marks, and finally generates meeting records by combining the handwriting marks, timestamps, and aligned meeting data. This integrates the handwriting marks made by the user during the meeting throughout the entire recording process, ensuring that the handwriting marks are reflected in the final generated meeting records.

Description

Technical Field

[0001] This application relates to the field of meeting recording technology, and more particularly to a multimodal meeting recording method, electronic device, storage medium, and program product. Background Technology

[0002] In fields such as knowledge work, business meetings, and teaching, the recording and review of information is a core element. While traditional smart pens or notebooks can use AI to take meeting minutes, they can only provide general summaries and cannot recognize handwritten markings (such as circle = focus, underline = delete / negation, exclamation mark = urgent, asterisk = important, question mark = doubt). Summary of the Invention

[0003] This application provides a multimodal conference recording method, electronic device, storage medium, and program product to at least solve one of the above-mentioned technical problems.

[0004] In a first aspect, embodiments of this application provide a multimodal meeting recording method, including: Simultaneously collect writing data and conference audio data, wherein the writing data includes writing trajectory data; Identify writing marks based on the writing trajectory data, and obtain the timestamps corresponding to the writing marks; The conference audio data is aligned by combining the writing marks and the writing data to obtain aligned conference data. Meeting minutes are generated based on the written markers, the marker timestamps, and the aligned meeting data.

[0005] In some embodiments, the writing marks include at least one of the following mark types: circle mark, underline, strikethrough, exclamation mark, asterisk, question mark, and arrow mark; identifying writing marks based on the writing trajectory data includes: inputting the writing trajectory data into a pre-trained writing mark recognition model to obtain writing marks.

[0006] In some embodiments, the writing data further includes pen-lifting intervals; the step of aligning the conference audio data by combining the writing marks and the writing data to obtain aligned conference data includes: The paragraph boundaries in the conference audio data are determined based on the writing marks, the pen lifting interval, and the duration of silence in the conference audio data.

[0007] In some embodiments, determining the paragraph boundaries in the conference audio data based on the writing mark, the pen lift interval, and the silence duration in the conference audio data includes: When the occurrence of the writing mark and the pen lifting interval both exceed the interval threshold, the segment boundary in the conference audio data is determined; or When the occurrence of the written mark and the duration of the silence exceed a time threshold are simultaneously satisfied, the segment boundary in the conference audio data is determined; or When both the pen lifting interval and the silence duration exceed the time threshold, the segment boundary in the conference audio data is determined.

[0008] In some embodiments, the writing data further includes pen-lifting action data; the step of combining the writing marks and the writing data to align the conference audio data to obtain aligned conference data further includes: The sentence boundary timestamps are determined based on the silence point timestamps in the meeting audio data and the pen-lifting action timestamps. A sentence-level timestamp index is established based on the sentence boundary timestamps.

[0009] In some embodiments, the step of aligning the conference audio data by combining the writing marks and the writing data to obtain aligned conference data further includes: Search the target sentence-level timestamp index into which the timestamp of the written mark falls; Obtain the target sentence corresponding to the target sentence-level timestamp index; The sentence that, after obtaining the writing mark, forms a complete semantic paragraph with the target sentence; The semantic paragraph timestamp index corresponding to the writing mark is determined based on the target sentence-level timestamp index and the sentence-level timestamp index of the subsequent sentence, and the semantic paragraph timestamp index corresponds to the complete semantic paragraph.

[0010] In some embodiments, generating meeting minutes based on the writing mark, the mark timestamp, and the aligned meeting data includes: Based on the marked timestamp, determine the semantic paragraph timestamp index corresponding to the written mark, and determine the complete semantic paragraph corresponding to the semantic paragraph timestamp index as the marked text content; Meeting minutes are generated based on the written markings and the content of the marked text.

[0011] In some embodiments, each of the tag types corresponds to a semantic tag; The step of generating meeting minutes based on the written marks and the marked text content includes: Based on the tag type to which the writing tag belongs, determine the semantic tag corresponding to the writing tag; Based on the semantic tags corresponding to the writing marks and the marked text content, prompt words are generated; The prompt words are input into a preset large language model to generate meeting minutes.

[0012] In some embodiments, when the writing mark includes a first writing mark and a second writing mark, and the first writing mark and the second writing mark correspond to the same mark text content, determining the semantic tag corresponding to the writing mark according to the mark type to which the writing mark belongs includes: Based on the tag type to which the first writing tag belongs, determine the first semantic tag corresponding to the first writing tag; Based on the tag type to which the second writing tag belongs, determine the second semantic tag corresponding to the second writing tag; Based on the first semantic tag and the second semantic tag, determine the semantic tag corresponding to the writing mark.

[0013] In some embodiments, when the first writing mark appears earlier than the second writing mark, determining the semantic label corresponding to the writing mark based on the first semantic label and the second semantic label includes: correcting the first semantic label using the second semantic label to determine the semantic label corresponding to the writing mark; when the first writing mark appears later than the second writing mark, determining the semantic label corresponding to the writing mark based on the first semantic label and the second semantic label includes: correcting the second semantic label using the first semantic label to determine the semantic label corresponding to the writing mark.

[0014] In some embodiments, the meeting minutes include at least one of the following: meeting summary, to-do list, and content analysis.

[0015] In some embodiments, the method further includes: fine-tuning a general large language model using fine-tuning data from a preset scenario to obtain the preset large language model.

[0016] In some embodiments, the method further includes: using OCR to recognize the text in the area marked by the writing mark.

[0017] In some embodiments, the method further includes: determining the confidence level of the writing notes based on the writing trajectory; and prompting the user to confirm when the confidence level is less than a confidence threshold.

[0018] Secondly, embodiments of this application provide a storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the multimodal conference recording methods described above.

[0019] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the multimodal conference recording methods described above in this application.

[0020] Fourthly, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to execute any of the above-described multimodal conference recording methods.

[0021] Fifthly, embodiments of this application also provide a multimodal conference recording system, which includes: The edge device is configured to simultaneously collect writing data and conference audio data. The writing data includes writing trajectory data. The device identifies writing marks based on the writing trajectory data and obtains the timestamps corresponding to the writing marks. The device then combines the writing marks and the writing data to align the conference audio data to obtain aligned conference data. A cloud-based device configured to generate meeting minutes based on the written markers, the marker timestamps, and the aligned meeting data.

[0022] This application synchronously collects written data and meeting audio data, identifies writing marks through writing trajectory data in the written data, aligns the meeting audio data with the writing marks, and finally generates meeting minutes by combining the writing marks, timestamps, and aligned meeting data. The writing marks made by the user during the meeting are integrated throughout the entire meeting recording process, so that the writing marks can be reflected in the final generated meeting minutes. This allows the meeting minutes to not only convey the content of the meeting itself, but also record the content that the user is particularly concerned about. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart of an embodiment of the multimodal meeting recording method of this application; Figure 2 A flowchart of another embodiment of the multimodal meeting recording method of this application; Figure 3 A flowchart of another embodiment of the multimodal meeting recording method of this application; Figure 4 A flowchart of another embodiment of the multimodal meeting recording method of this application; Figure 5 A flowchart of another embodiment of the multimodal meeting recording method of this application; Figure 6 A flowchart of another embodiment of the multimodal meeting recording method of this application; Figure 7 This is a schematic block diagram of one embodiment of the multimodal recording system of this application; Figure 8 This is a flowchart illustrating an embodiment of the multimodal meeting recording system of this application; Figure 9 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0026] It should also be noted that, in this document, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0027] like Figure 1 As shown, an embodiment of this application provides a multimodal meeting recording method, including: S10. Synchronously collect writing data and conference audio data, wherein the writing data includes writing trajectory data.

[0028] For example, the above steps can be performed using devices such as smart pens, notebooks, and tablets, where notebooks and tablets can be equipped with electromagnetic pens or styluses. Smart pens can perform both writing and recording; notebooks can be used for writing with electromagnetic pens and for recording with their built-in microphones; tablets can be used for writing with styluses and for recording with their built-in microphones. For example, after synchronously acquiring writing data and meeting audio data, the data is cached on local storage media and selectively synchronized to the cloud when a network is available.

[0029] In some embodiments, the smart pen can also be used with an external terminal device. This external terminal can be a smartphone, tablet, laptop, or office notebook, etc., and this application does not limit this to any particular device. The following embodiments only use the smart pen in conjunction with a smartphone as an example.

[0030] In some embodiments, a precise time base at the hardware level ensures that the sampling times of written data and conference audio data are strictly aligned. For example, a hardware clock is used: a 32.768kHz crystal oscillator with an accuracy of ±20ppm.

[0031] Synchronization mechanisms are as follows: 1. At the start of the meeting, the smart pen and mobile phone synchronize their clocks (NTP or Bluetooth clock synchronization). 2. Collection of each handwriting point: Record the timestamp T_pen (milliseconds); 3. Audio frame capture: Record timestamp T_audio (milliseconds); 4. Error control: |T_pen - T_audio| < 5ms.

[0032] The specific implementation is as follows: - Handwriting sampling: 100Hz (one sampling point every 10ms); - Audio sampling: 16kHz (one sample point every 0.0625ms); - Unified Clock: The smart pen's MCU maintains a global timer, and the interrupt service routine reads writing data and conference audio data simultaneously.

[0033] S20. Identify writing marks based on the writing trajectory data, and obtain the timestamp corresponding to the writing marks. The timestamp is the time the user drew the writing marks.

[0034] For example, the writing marks include at least one of the following mark types: circle mark, underline, strikethrough, exclamation mark, asterisk, question mark and arrow mark; identifying the writing marks based on the writing trajectory data includes: inputting the writing trajectory data into a pre-trained writing mark recognition model to obtain the writing marks.

[0035] For example, the writing trajectory data includes a sequence of dot coordinates of a smart pen (or a sequence of writing trajectory points of an electromagnetic pen / stylus). The writing mark recognition model employs a lightweight CNN (e.g., MobileNetV3). For example, inputting the writing trajectory data into a pre-trained writing mark recognition model to obtain writing marks can be achieved through the following steps: Input: Write the sequence of trajectory points [(x1,y1,t1), (x2,y2,t2), ..., (xn,yn,tn)]; CNN processing: The lightweight MobileNetV3 network extracts trajectory shape features from the sequence of written trajectory points; Output: Tag type + confidence (e.g., type="circle", confidence=0.95).

[0036] S30. Align the conference audio data by combining the writing marks and the writing data to obtain aligned conference data. For example, the paragraph boundaries in the conference audio data are determined based on the writing marks, the pen lifting interval, and the silence duration in the conference audio data.

[0037] S40. Generate meeting minutes based on the writing marks, the mark timestamps, and the aligned meeting data. For example, text data corresponding to the mark data is obtained from the aligned meeting data based on the mark timestamps, and then prompt words are generated based on the writing marks and text data and input into a preset large language model to generate meeting minutes. The meeting minutes include at least one of the following: meeting summaries, to-do lists, and content analysis.

[0038] This application synchronously collects written data and meeting audio data, identifies writing marks through writing trajectory data in the written data, aligns the meeting audio data with the writing marks, and finally generates meeting minutes by combining the writing marks, timestamps, and aligned meeting data. The writing marks made by the user during the meeting are integrated throughout the entire meeting recording process, so that the writing marks can be reflected in the final generated meeting minutes. This allows the meeting minutes to not only convey the content of the meeting itself, but also record the content that the user is particularly concerned about.

[0039] In some embodiments, determining the paragraph boundary in the conference audio data based on the writing mark, the pen lift interval, and the silence duration in the conference audio data includes: determining the paragraph boundary in the conference audio data when the occurrence of the writing mark and the pen lift interval both exceed an interval threshold; or determining the paragraph boundary in the conference audio data when the occurrence of the writing mark and the silence duration both exceed a time threshold; or determining the paragraph boundary in the conference audio data when the pen lift interval and the silence duration both exceed an interval threshold.

[0040] For example, aligning the conference audio data by combining the writing marks and the writing data to obtain aligned conference data can be achieved as follows: 1. Extract writing features: pen lift interval Δt_pen; 2. Extract speech features: Detect the duration of silence in speech; 3. Joint detection of paragraph boundaries: - Condition 1: Δt_pen > interval threshold (writing pause); - Condition 2: Duration of voice silence > time threshold (voice pause); - Condition 3: The marker symbol appears (semantic boundary); 4. If any two of conditions 1, 2 and 3 are satisfied, it is determined to be a paragraph boundary.

[0041] In some embodiments, the multimodal meeting recording method further includes: determining the confidence level of the written notes based on the writing trajectory; and prompting the user to confirm when the confidence level is less than a confidence threshold.

[0042] like Figure 2 The diagram shows a flowchart of another embodiment of the multimodal meeting recording method of this application. In this embodiment, the writing data further includes pen-lifting action data; the step of aligning the meeting audio data by combining the writing marks and the writing data to obtain aligned meeting data further includes: S301. Determine the sentence boundary timestamp based on the silence point timestamp in the conference audio data and the pen lifting action timestamp.

[0043] S302. Establish a sentence-level timestamp index for the corresponding sentence based on the sentence boundary timestamp.

[0044] For example, regarding the ASR result of meeting audio data: "Our Q2 revenue increased by 30% [1200ms-2500ms], but user retention decreased by 5% [2600ms-4000ms]", the alignment process in this embodiment includes: 1. Voice VAD detects silence points (a 200ms pause at 2600ms, i.e., a speech pause) → sentence boundary, i.e., the silence point timestamp is 2600ms; 2. Handwriting detection: There is a pen lifting action (i.e., writing pause) between 2500ms and 2700ms → writing boundary, that is, the pen lifting action timestamp is [2500ms-2700ms]; 3. Joint determination: speech pause + writing pause → confirm sentence boundaries; 4. Create sentence-level time indexes: sentence1=[1200,2500], sentence2=[2600,4000].

[0045] This embodiment implements sentence-level alignment of ASR results corresponding to conference audio data.

[0046] like Figure 3 The diagram shows a flowchart of another embodiment of the multimodal conference recording method of this application. In this embodiment, the step of aligning the conference audio data by combining the writing marks and the writing data to obtain aligned conference data further includes: S311. Search the target sentence-level timestamp index into which the timestamp of the written mark falls; S312. Obtain the target sentence corresponding to the target sentence-level timestamp index; S313. Obtain the sentence that, after the writing mark, forms a complete semantic paragraph with the target sentence; S314. Determine the semantic paragraph timestamp index corresponding to the writing mark based on the target sentence-level timestamp index and the sentence-level timestamp index of the subsequent sentence, wherein the semantic paragraph timestamp index corresponds to the complete semantic paragraph.

[0047] For example, in the scenario where a user circles "retention rate decreased by 5%" (timestamp = 3000ms), the alignment process in this embodiment includes: 1. A marker was detected (circle@3000ms); 2. Forward search: Find the sentence containing 3000ms (sentence 2 = [2600, 4000]); 3. Extending backwards: After marking, the voice continues to say "This needs special attention" [4000ms-5000ms]; 4. Semantic paragraph = [2600, 5000ms], containing: - The circled text reads: "Retention rate decreased by 5%". - Explanation after the mark: "Requires close attention" 5. This paragraph is used as an independent unit to input the preset LLM (i.e., preset large language model) for analysis, generating key insights.

[0048] This embodiment achieves semantic paragraph-level alignment of the ASR results corresponding to the conference audio data, driven by writing marks. By forcibly triggering paragraph boundaries through the appearance of writing marks, it ensures that the complete semantic units of interest to the user are not fragmented.

[0049] like Figure 4 The diagram shows a flowchart of another embodiment of the multimodal meeting recording method of this application. In this embodiment, generating meeting minutes based on the writing markers, the marker timestamps, and the aligned meeting data includes: S41. Based on the marker timestamp, determine the semantic paragraph timestamp index corresponding to the writing marker, and determine the complete semantic paragraph corresponding to the semantic paragraph timestamp index as the marked text content. For example, after determining the writing marker and its corresponding timestamp, the semantic paragraph timestamp index to which the timestamp falls is determined; then the complete semantic paragraph corresponding to that semantic paragraph timestamp index is the marked text content corresponding to the current writing marker.

[0050] S42. Generate meeting minutes based on the written marks and the marked text content.

[0051] In this embodiment, when determining the marked text content corresponding to the written mark, the semantic paragraph timestamp index of the complete semantic paragraph determined in the previous embodiment is used. That is, by directly searching through the mark timestamp corresponding to the written mark, the semantic paragraph timestamp index that it falls into can be retrieved, thereby quickly determining the complete semantic paragraph as the marked text content. This not only ensures the semantic integrity of the marked text content corresponding to the written mark, but also improves the efficiency of determining the marked text content.

[0052] like Figure 5 The diagram shows a flowchart of another embodiment of the multimodal meeting recording method of this application. In this embodiment, each of the marker types corresponds to a semantic tag; the step of generating meeting minutes based on the written markers and the marked text content includes: S421. Determine the semantic label corresponding to the writing mark based on its mark type. For example, a CNN is used to identify the mark type of the writing trajectory data. Each mark type corresponds to a semantic label, as shown in the table below:

[0053] The table above also shows that different marker types have different graphic features, and CNNs can identify marker types based on these features. Furthermore, different marker types can trigger different AI behaviors.

[0054] S422. Generate prompt word content based on the semantic tag corresponding to the writing mark and the mark text content.

[0055] For example, construct a "tag-text" association matrix: M[i, j], which represents the association weight between written tag i and tag text content j. Then, generate different conditional prompts for different tag types: - For the selected area: "...Please focus on analyzing the decision-making basis for the following content..." - For exclamation mark markers: "...mark the following as high priority tasks..." - For question mark markers: "...identify the questions below and provide suggested answers..." The corresponding marked text content is concatenated after different conditional prompts and input into the preset large language model along with the corresponding association matrix.

[0056] S423. Input the prompt words into a preset large language model to generate meeting minutes.

[0057] For example, the preset large language model (i.e., the general LLM) is obtained by fine-tuning the general large language model using fine-tuning data of a preset scenario. The general LLM can be, for example, GPT-4, Wenxin Yiyan, or Tongyi Qianwen, and this application does not limit it.

[0058] For example, in a meeting scenario: fine-tuning data = meeting minutes sample, enhancing decision extraction and task generation; For example, in an educational setting: fine-tuning data = classroom notes samples, strengthening the review of knowledge points and answering questions; For example, in the medical setting: fine-tuning data = medical record samples, strengthening the extraction of medical orders and the correlation of diagnoses.

[0059] Fine-tuning methods: - LoRA low-rank adaptation, frozen general LLM, training domain adapter; - Prompt Engineering: Preset scenario-specific System Prompt At runtime: - Input: General LLM + Domain Adapter + Dynamic Condition Hints; - Output: Structured analysis results of scene optimization.

[0060] In the process of developing this application, the inventors discovered that during actual meetings, users may continuously draw multiple writing marks while taking notes. For example, a user might continuously draw: a circle → a strikethrough → an exclamation mark. This can be categorized into two scenarios: either the three writing marks pertain to different content, or at least two of the writing marks pertain to the same content. For the first scenario, the aforementioned embodiments can be used to process each writing mark distribution individually. For the second scenario, the method described in this embodiment is used to determine a unique semantic tag.

[0061] In some embodiments, methods for determining whether two written marks correspond to the same marked text content include, but are not limited to, the following: determining whether two written marks overlap in spatial position (e.g., determining whether the corresponding mark intervals of two written marks at least partially overlap each other).

[0062] like Figure 6 The diagram shows a flowchart of another embodiment of the multimodal meeting recording method of this application. In this embodiment, when the writing mark includes a first writing mark and a second writing mark, and the first writing mark and the second writing mark correspond to the same marked text content, determining the semantic tag corresponding to the writing mark according to the mark type to which the writing mark belongs includes: S4211. Determine the first semantic label corresponding to the first writing mark based on the mark type to which the first writing mark belongs. For example, if the first writing mark is "circle", then the corresponding first semantic label is "area of ​​focus".

[0063] S4212. Determine the second semantic tag corresponding to the second writing mark based on the mark type to which the second writing mark belongs. For example, if the second writing mark is "strikethrough", then the corresponding second semantic tag is "negation / obsolete".

[0064] S4213. Determine the semantic tag corresponding to the writing mark based on the first semantic tag and the second semantic tag.

[0065] For example, based on the first semantic label "key area of ​​concern" and the second semantic label "negation / abandonment", a new semantic label "this key area has been resolved" can be determined, so the pre-defined large language model will no longer focus on parsing this area.

[0066] For example, when the first writing mark appears earlier than the second writing mark, determining the semantic label corresponding to the writing mark based on the first semantic label and the second semantic label includes: correcting the first semantic label with the second semantic label to determine the semantic label corresponding to the writing mark; when the first writing mark appears later than the second writing mark, determining the semantic label corresponding to the writing mark based on the first semantic label and the second semantic label includes: correcting the second semantic label with the first semantic label to determine the semantic label corresponding to the writing mark.

[0067] The above embodiments can employ Transformer, for example: The user drew the following consecutively: circle → strikethrough → exclamation mark, strikethrough → circle → exclamation mark, or asterisk → question mark; >- CNN recognition: three independent markers (circle, strikethrough, exclamation) or two independent markers; Transformer processing: analyzing sequence relationships; > - For overlapping "circle + strikethrough", and drawing the circle first and then the strikethrough = marking the important part first, then negating it (may indicate "this important part has been resolved"); > - For overlapping strikethrough and circle, drawing the strikethrough first and then the circle = marking the important part first and then negating it (which may indicate "the content to be deleted needs to be given special attention"); > - The overlapping of "strikethrough + exclamation mark" = negation but urgency (may indicate "although negated, it still needs attention"); > - For issues where "asterisk + question mark" overlap, it indicates a key area of ​​focus; >- Output: Corrected semantic labels (i.e., "This focus has been addressed" or "Although it is negative, it still needs attention").

[0068] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0069] like Figure 7As shown, embodiments of this application also provide a multimodal meeting recording system 700, including: a handwritten semantic tag recognition engine 710, a multimodal alignment engine 720, and a tag-driven AI analysis engine 730. The three engines are described below: Handwritten semantic markup recognition engine 710: Unlike existing OCR systems that only recognize text, this handwritten semantic mark recognition engine specifically identifies functional marker symbols. For example, the technical implementation is as follows: • Input: Dot coordinate sequence (smart pen) or handwritten trajectory point sequence (electromagnetic pen / stylus pen); • Processing: Lightweight CNN (MobileNetV3) for symbol classification + Transformer for temporal context modeling; • Output: Tag type + Spatial location (bounding box) + Timestamp + Confidence.

[0070] The marker types include selection circles, underlines, strikethroughs, exclamation marks, asterisks, question marks, and arrows.

[0071] Spatial location refers to the bounding box coordinates of the writing mark, which can be determined based on raster coordinates or pixel coordinates. For example, the minimum bounding rectangle of the writing mark: bbox=[min(x), min(y), max(x), max(y)], is used to associate with OCR text to determine the text corresponding to the writing mark. For example, a user circles "Q2 revenue increased by 30%": >- Trajectory points: (120,80), (125,82), ..., (400,120) >- Spatial location: bbox="120,80,400,120" >- Related OCR: The text within this bbox = "Q2 revenue growth of 30%".

[0072] For example, the smart pen includes a miniature camera module for capturing images of paper and using OCR technology to recognize the written content to obtain OCR text; the miniature camera module supports autofocus, is suitable for shooting documents at close range of 5-10cm, and has a resolution of no less than 300dpi.

[0073] A timestamp indicates the time when a written mark appears. It is used to align with conference audio data to find the audio content at the time of the written mark; it can also be used to align with ASR text to find the text corresponding to the mark; and it can also be used for time-series sorting of multiple marks, etc.

[0074] Confidence level reflects the reliability of tag type identification. The functions of confidence level are as follows: 1. Threshold filtering: When confidence < 0.8, the marker is treated as normal writing; 2. Weight Calculation: High-confidence tags are assigned higher weights for AI analysis; 3. User feedback: Prompt users to confirm when confidence is low; For example: Recognition result: type="circle", confidence=0.6 (low); >- Processing strategy: The marking may not be intentional; the trajectory is simply recorded without triggering AI analysis. >- Or prompt the user: "Would you like to mark this as important?".

[0075] Multimodal Alignment Engine 720: Existing solutions only perform coarse-grained timestamp alignment. This application uses a multimodal alignment engine to achieve joint detection of writing pauses and speech pauses. The alignment algorithm flow is as follows: 1. Extract writing features: pen speed v(t), pressure p(t), pen lift interval Δt_pen; 2. Extract speech features: energy E(t), silence detection, speaker switching points; 3. Joint detection of paragraph boundaries: - Condition 1: Δt_pen > threshold (writing pause); - Condition 2: Duration of speech silence > threshold (speech pause); - Condition 3: The marker symbol appears (semantic boundary); 4. If any two of conditions 1+2+3 are met, it is determined to be a paragraph boundary; 5. Establish a three-level time index: - L1: Millisecond-level raw synchronization point (hardware clock); - L2: Sentence-level alignment (ASR results); - L3: Semantic paragraph level (handwritten markup driven).

[0076] The semantic boundary in the above embodiments means the logical paragraph dividing point of the meeting content, that is, the moment of topic change or focus shift.

[0077] Tag-driven AI analytics engine 730: Traditional workflow: Voice → ASR → General LLM summary → Output minutes; The label-driven AI analysis engine process described in this application is as follows: speech → ASR → handwritten label semantic injection → conditional prompting engineering → domain LLM analysis → structured output.

[0078] "Handwritten markup semantic injection" includes: 1. Map the writing mark positions to the ASR text timeline; 2. Construct a "tag-text" association matrix: M[i,j] = the association weight between tag i and word j; 3. Generate conditional prompts: - For the selected area: "...Please focus on analyzing the decision-making basis for the following content..."; - For exclamation mark markers: "...mark the following as high priority pending tasks..."; - For the question mark marker: "...identify the points of confusion regarding the following and provide suggested answers..."; In some embodiments, the tag-driven AI analysis engine includes three sub-modules: an intent understanding engine (sub-module 1), semantic fusion analysis (sub-module 2), and conditional prompt generation (sub-module 3). Specifically, the intent understanding engine: tags semantics → AI behavior instructions, e.g., circle + focus → "deep_analysis" instruction; semantic fusion analysis: multimodal data → structured results, e.g., text + tags → key summary; conditional prompt generation: dynamically constructing LLM input, e.g., selecting different prompt templates based on tag type.

[0079] In some embodiments, this application employs a unified markup semantic intermediate format to implement cross-device collaborative protocols. For example, the unified markup semantic intermediate format (Handwriting Markup Language, HML) is defined as follows: <session id="mtg_20260317" device="P100_v1"><!--Session identifier: Meeting ID is < / session> mtg_20260317, device is the first-generation Smart Pen P100--> <track type="audio" codec="opus" sample_rate="16000"> <!--Audio track: Opus encoding, 16 kHz sampling rate--> <track type="ink" resolution="600dpi" technology="AISPEECH"> <!--Pen Trace track: 600 dpi accuracy, AISPEECH technology--> <mark type="circle" timestamp="1234567" bbox="x1,y1,x2,y2" <!--Marker Definition: Type is circle, timestamp 1234567 ms, bounding box coordinates x1, y1, x2, y2--> semantic="focus" confidence="0.95"> <!--Semantic is "focus", recognition confidence 95% --> <linked_audio start="1234000" end="1235000" / > <!--Linked audio segment: Start time 1234000 ms, end time 1235000 ms--> <linked_text>OCR recognition content< / linked_text> <!--Linked text: Text content recognized by OCR within the marked area--> <ai_action type="deep_analysis" priority="high" / > <!--AI action instruction: Type is deep analysis, priority is high--> Figure 8 The XML code above defines a complete multimodal data record of a conference session, including audio configuration, handwriting configuration, and a specific "circle" marker and its associated audio segment, OCR text, and AI analysis instructions, achieving semantic-level association between handwritten markers and speech content. An optional data flow example is as follows: Circle on paper → Smart pen acquisition → HML formatting → Bluetooth transmission → Mobile APP parsing → Displaying the circle's position, associating it with the audio segment, triggering in-depth analysis → Generating a key analysis report.

[0080] like Figure 9 The diagram shown is a flowchart of the multimodal conference recording system of this application. This embodiment includes the following steps: S1. The edge device layer (including smart pens, notebooks, Pads / tablets, and other forms) simultaneously collects writing data and conference audio data; S2. The edge device layer preprocesses the written data and conference audio data through the local preprocessing module configured in the edge computing chip; S3, the edge device layer performs handwritten data caching, audio stream caching, and tag semantic caching, etc.; among them, the functions of tag semantic caching include: - Temporarily stores the type of markings (e.g., writing marks) recognized in real time by the pen tip (e.g., smart pen, electromagnetic pen, or stylus). - Fast response: As soon as a user finishes drawing a writing mark, it is immediately cached locally without waiting for the complete transmission; - Driving subsequent processes: The cached symbol type directly triggers two actions: 1. Real-time transmission: Instantly send data to the mobile app for display via Bluetooth; 2. Alignment Engine: Serves as a "semantic boundary" signal and participates in multimodal alignment determination.

[0081] Specific examples: The user drew a circle (marker type = circle), and the pen-based CNN recognized it as follows: >- Write to cache immediately: "cache value = circle"; >- Trigger Bluetooth transmission: The phone immediately displays the position of the "circle" on the paper; >- Trigger the alignment engine: Determine if "circle appears" = semantic paragraph boundary.

[0082] S4. Employ a multimodal alignment engine (to achieve timestamp + semantic alignment). S5. Then, based on the data type and / or network status, automatically switch between a first transmission mode (real-time low-power transmission, such as Bluetooth transmission) and a second transmission mode (high-speed batch transmission, such as WiFi transmission). For example, aligned data is transmitted to the cloud intelligent layer through real-time transmission, batch synchronization, and offline enhancement. Specifically, real-time transmission transmits urgent and small data such as tagged event packets to the cloud intelligent layer; for example, Bluetooth low-latency transmission allows the phone to immediately display the tagged location, providing real-time user feedback. Batch synchronization transmits large audio files (large data, non-urgent) to the cloud intelligent layer; for example, WiFi transmission of audio files accumulated after a meeting, enabling high-speed and complete synchronization. Offline enhancement stores locally cached data, storing it locally when there is no network and automatically re-transmitting it after the network is restored, ensuring no data loss. For example: During the 2-hour meeting: >- Draw a ☆ mark → Real-time transmission (Bluetooth, 50ms delay, displayed on the phone immediately); >- Continuous recording → local caching, meeting end → batch synchronization (WiFi); >- Bluetooth disconnection during meeting → Offline enhancement (save in pen, automatically re-upload tags after Bluetooth is restored).

[0083] S6. The received data is processed through a handwritten symbol recognition module, an automatic speech recognition (ASR) module, and a semantic fusion analysis module (multimodal LLM) to perform symbol recognition, automatic speech recognition, and semantic fusion. For example, the semantic fusion analysis module functions as follows: - Input data fusion: ASR text + handwritten mark semantics + spatiotemporal correlation matrix are uniformly encoded into the large model; - Analysis and processing: Differentiated summarization of speech content based on labeled semantic weights; - Output: Structured meeting minutes (key paragraphs / to-do items / question list).

[0084] For example: - Input data: ├─ ASR text: "Our Q2 revenue grew by 30%, but user retention decreased by 5%, which needs to be closely monitored." ├─ Semantic Markup: The user drew a "circle" on "5% decrease in retention" (semantic="focus"); └─ Spatiotemporal correlation: The timestamp of the circle = 1234567ms, corresponding to words 15-20 of the ASR text; -Processing procedure: 1. Construct an association matrix: M[cycles, "retention decline"] = high weight; 2. Generating conditional prompts: "Please focus on analyzing the decision-making basis and risks of '5% decrease in retention'"; 3. Multimodal LLM analysis: Utilizing contextual information to generate in-depth insights into the problem; - Output results: ├─ Key Section: Detailed analysis of the "declining retention" section selected by users; ├─ To-do list: Propose 3 improvement suggestions to address retention issues; └─ Question List: Should we adjust our Q3 user operation strategy? S7. The intent understanding engine maps labeled semantics to AI behavioral instructions. For example, labeled semantics → AI behavioral instructions (circle + focus → “deep_analysis” instruction).

[0085] S8, the meeting minutes generation module generates structured meeting minutes; the to-do extraction engine outputs to-do items according to priority; the content insight analysis module performs key analysis on content of interest.

[0086] In some embodiments, this application also provides a multimodal conference recording system, which includes: The edge device is configured to simultaneously collect writing data and conference audio data. The writing data includes writing trajectory data. The device identifies writing marks based on the writing trajectory data and obtains the timestamps corresponding to the writing marks. The device then combines the writing marks and the writing data to align the conference audio data to obtain aligned conference data. A cloud-based device configured to generate meeting minutes based on the written markers, the marker timestamps, and the aligned meeting data.

[0087] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the multimodal conference recording methods described above.

[0088] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described multimodal conference recording methods.

[0089] In some embodiments, this application also provides an electronic device including: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a multimodal conference recording method.

[0090] Figure 9 This is a schematic diagram of the hardware structure of an electronic device for performing a multimodal conference recording method according to another embodiment of this application, as shown below. Figure 9 As shown, the device includes: One or more processors 910 and memory 920, Figure 9 Take the 910 processor as an example.

[0091] The device for performing the multimodal conference recording method may further include an input device 930 and an output device 940.

[0092] The processor 910, memory 920, input device 930, and output device 940 can be connected via a bus or other means. ​ Taking the example of a connection between China and Israel via a bus.

[0093] The memory 920, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the multimodal conference recording method in the embodiments of this application. The processor 910 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 920, thereby implementing the multimodal conference recording method in the above-described method embodiments.

[0094] The memory 920 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the multimodal conference recording method apparatus. Furthermore, the memory 920 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 920 may optionally include memory remotely located relative to the processor 910, and these remote memories may be connected to the multimodal conference recording method apparatus via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] Input device 930 can receive input digital or character information and generate signals related to user settings and function control of the multimodal conference recording method apparatus. Output device 940 may include display devices such as a display screen.

[0096] The one or more modules are stored in the memory 920, and when executed by the one or more processors 910, they execute the multimodal conference recording method in any of the above method embodiments.

[0097] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0098] The electronic devices in this application embodiments exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0099] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0100] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0101] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0102] (5) Other electronic devices with data interaction functions.

[0103] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for recording multimodal meetings, comprising: Simultaneously collect writing data and conference audio data, wherein the writing data includes writing trajectory data; Identify writing marks based on the writing trajectory data, and obtain the timestamps corresponding to the writing marks; The conference audio data is aligned by combining the writing marks and the writing data to obtain aligned conference data. Meeting minutes are generated based on the written markers, the marker timestamps, and the aligned meeting data.

2. The method according to claim 1, characterized in that, The writing marks include at least one of the following mark types: circle mark, underline, strikethrough, exclamation mark, asterisk, question mark and arrow mark; Identifying writing marks based on the writing trajectory data includes: inputting the writing trajectory data into a pre-trained writing mark recognition model to obtain writing marks.

3. The method according to claim 2, characterized in that, The writing data also includes pen-lifting intervals; the process of aligning the conference audio data by combining the writing marks and the writing data to obtain aligned conference data includes: The paragraph boundaries in the conference audio data are determined based on the writing marks, the pen lifting interval, and the duration of silence in the conference audio data.

4. The method according to claim 3, characterized in that, Determining the paragraph boundaries in the conference audio data based on the writing marks, the pen lifting interval, and the silence duration in the conference audio data includes: When the occurrence of the writing mark and the pen lifting interval both exceed the interval threshold, the segment boundary in the conference audio data is determined; or When the occurrence of the written mark and the duration of the silence exceed a time threshold are simultaneously satisfied, the segment boundary in the conference audio data is determined; or When both the pen lifting interval and the silence duration exceed the time threshold, the segment boundary in the conference audio data is determined.

5. The method according to claim 3, characterized in that, The writing data also includes pen-lifting action data; the process of aligning the conference audio data by combining the writing marks and the writing data to obtain aligned conference data further includes: The sentence boundary timestamps are determined based on the silence point timestamps in the meeting audio data and the timestamp of the pen-lifting action. A sentence-level timestamp index is established based on the sentence boundary timestamps.

6. The method according to claim 5, characterized in that, The step of aligning the conference audio data by combining the writing marks and the writing data to obtain aligned conference data further includes: Search the target sentence-level timestamp index into which the timestamp of the written mark falls; Obtain the target sentence corresponding to the target sentence-level timestamp index; The sentence that, after obtaining the writing mark, forms a complete semantic paragraph with the target sentence; The semantic paragraph timestamp index corresponding to the writing mark is determined based on the target sentence-level timestamp index and the sentence-level timestamp index of the subsequent sentence, and the semantic paragraph timestamp index corresponds to the complete semantic paragraph.

7. The method according to claim 6, characterized in that, The step of generating meeting minutes based on the writing mark, the mark timestamp, and the aligned meeting data includes: Based on the marked timestamp, determine the semantic paragraph timestamp index corresponding to the written mark, and determine the complete semantic paragraph corresponding to the semantic paragraph timestamp index as the marked text content; Meeting minutes are generated based on the written markings and the content of the marked text.

8. The method according to claim 7, characterized in that, Each of the aforementioned tag types corresponds to a semantic tag; The step of generating meeting minutes based on the written marks and the marked text content includes: Based on the tag type to which the writing tag belongs, determine the semantic tag corresponding to the writing tag; Based on the semantic tags corresponding to the writing marks and the marked text content, prompt words are generated; The prompt words are input into a preset large language model to generate meeting minutes.

9. The method according to claim 8, characterized in that, When the writing mark includes a first writing mark and a second writing mark, and the first writing mark and the second writing mark correspond to the same mark text content, determining the semantic tag corresponding to the writing mark according to the mark type to which the writing mark belongs includes: Based on the tag type to which the first writing tag belongs, determine the first semantic tag corresponding to the first writing tag; Based on the tag type to which the second writing tag belongs, determine the second semantic tag corresponding to the second writing tag; Based on the first semantic tag and the second semantic tag, determine the semantic tag corresponding to the writing mark.

10. The method according to claim 9, characterized in that, When the first writing mark appears before the second writing mark, the semantic label corresponding to the writing mark is determined based on the first semantic label and the second semantic label, including: using the second semantic label to correct the first semantic label, and then determining the semantic label corresponding to the writing mark; When the first writing mark appears later than the second writing mark, the semantic label corresponding to the writing mark is determined based on the first semantic label and the second semantic label, including: using the first semantic label to correct the second semantic label, and then determining the semantic label corresponding to the writing mark.

11. The method according to claim 8, characterized in that, The meeting minutes include at least one of the following: meeting summary, to-do list, and content analysis.

12. The method according to claim 8, characterized in that, Also includes: The preset large language model is obtained by fine-tuning the general large language model using fine-tuning data from a preset scenario.

13. The method according to any one of claims 1-12, characterized in that, Also includes: The text in the area marked by the written mark is recognized by OCR.

14. The method according to any one of claims 1-12, characterized in that, The method further includes: determining the confidence level of the writing notes based on the writing trajectory; When the confidence level is less than the confidence threshold, the user is prompted to confirm.

15. A multimodal conference recording system, characterized in that, include: The edge device is configured to simultaneously collect writing data and conference audio data. The writing data includes writing trajectory data. The device identifies writing marks based on the writing trajectory data and obtains the timestamps corresponding to the writing marks. The device then combines the writing marks and the writing data to align the conference audio data to obtain aligned conference data. A cloud-based device configured to generate meeting minutes based on the written markers, the marker timestamps, and the aligned meeting data.

16. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-14.

17. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-14.

18. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-14.

Images