Multimodal conference data synchronization method, system, electronic device, storage medium and program product
By configuring event identifiers and establishing mapping relationships for the writing data and audio data of the smart pen, the problem of synchronizing the writing trajectory and audio data in the smart pen is solved, achieving an efficient and accurate note playback experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- AISPEECH CO LTD
- Filing Date
- 2026-04-13
- Publication Date
- 2026-07-14
AI Technical Summary
In the current smart pen, the writing trajectory and audio data synchronization during the recording process have strong time base dependence, high coupling of multi-stream data, large amount of data redundancy and complex playback timing, which makes it impossible for users to quickly obtain the corresponding audio content of the notes, affecting the integrity and efficiency of information.
By configuring associated writing event identifiers and audio event identifiers for writing data and establishing a mapping relationship, the writing data and audio data are stored, and a counter is used to configure a monotonically increasing sequence number for the data, thus avoiding dependence on the accuracy and stability of the RTC clock.
It achieves efficient synchronization of written data and audio data, simplifies the playback process, and improves the efficiency and completeness of information retrieval when users review their notes.
Smart Images
Figure CN122394981A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of meeting recording technology, and in particular to a multimodal meeting data synchronization method, system, electronic device, storage medium, and program product. Background Technology
[0002] In fields such as knowledge work, business meetings, and education, the recording and review of information is a core and ongoing process. Whether students are attending lectures, journalists are conducting interviews, or professionals are participating in project discussions, taking notes with pen and paper, supplemented by audio recordings, is an extremely common and efficient method.
[0003] As a new type of electronic device, smart pens can digitally capture a user's handwriting and convert it into digital information usable in various applications. For example, in optical-based smart pens, built-in optical sensors can detect and record the precise coordinates of the pen tip relative to a digitally encoded surface (such as dot-matrix paper), thus completely reproducing the writing trajectory. More importantly, many smart pens also integrate microphones, enabling simultaneous audio recording while capturing handwriting. This synchronized audio and gesture data retains the natural feel and convenience of paper writing while incorporating the powerful features of digital environments, such as searchability and playback capabilities, providing users with a richer note-taking experience that transcends traditional pen and paper.
[0004] However, as note-taking time increases and content accumulates, the smart pen collects a vast amount of information, including dozens of pages of writing traces and hours of audio recordings, which are stored on the device or in the cloud. If the writing trace data and audio recording data are not effectively synchronized and correlated, users will not be able to quickly access the corresponding audio content when reviewing the explanations in a particular section of notes, making it difficult to guarantee the completeness of the information obtained.
[0005] Therefore, how to accurately synchronize the writing trajectory and audio data collected by the smart pen during the recording process, so as to ensure that users can achieve an efficient, accurate and complete review experience, has become a key challenge that urgently needs to be solved in the development of smart pen technology.
[0006] To address this issue, related technologies employ smart pens that synchronize writing capture by collecting positional and temporal sample data (x, y, t) (where x and y are the coordinates of the smart pen in the plane during writing, and t is the timestamp corresponding to those coordinates). Simultaneously, the recorded audio is also sampled with reference to the same time t. Alternatively, the smart pen establishes a unified time index at the start of a recording session, with all three data streams (writing data, audio data, and digital content browsing data) recorded and sampled using the same timeline. However, these related technologies suffer from the following problems: 1. High dependence on time base: The system is highly dependent on the accuracy and stability of the RTC clock; clock drift will lead to the accumulation of synchronization errors. 2. High coupling of multi-stream data: Written data, audio data, and digital content browsing data must be strictly aligned to the same timeline; any abnormality in any data stream will affect the overall synchronization. 3. High data redundancy: Continuous recording of high-precision timestamps increases storage overhead; 4. Complex playback timing: It requires maintaining a global timeline, making cross-session data association difficult; 5. Unified timeline dependency: A globally unified time index must be established to force multi-stream data to be bound to the same timeline. Summary of the Invention
[0007] This application provides a multimodal conference data synchronization method, system, electronic device, storage medium, and program product to at least solve one of the above-mentioned technical problems.
[0008] In a first aspect, embodiments of this application provide a method for synchronizing multimodal conference data, comprising: Detect writing events and configure writing event identifiers for the corresponding writing data; When the writing event is detected, an audio event is detected, and corresponding audio data is collected and an audio event identifier is configured for the corresponding audio data; Establish a mapping relationship between the writing event identifier and the audio event identifier, and store the writing data and the corresponding audio data.
[0009] In some embodiments, the writing event identifier is a writing event sequence number, and the audio event identifier is an audio event sequence number. The writing event sequence number and the audio event sequence number exhibit a monotonically increasing trend according to the chronological order in which the writing data and the corresponding audio data occur. The method further includes: A counter is used to configure the writing event sequence number for the written data; A counter is used to configure the audio event sequence number for the corresponding audio data.
[0010] In some embodiments, the writing data includes at least one writing segment, and the corresponding audio data includes at least one audio segment.
[0011] In some embodiments, the counter is configured with the writing event sequence number and / or the audio event sequence number for the writing data and / or the corresponding audio data in an atomic increment manner.
[0012] In some embodiments, the writing data includes a first writing segment and a second writing segment that occur sequentially; configuring the writing event sequence number for the writing data using a counter includes: The first writing event sequence number is configured for the first writing segment using the counter; The counter is used to configure a second writing event sequence number for the second writing segment; The first writing event sequence number is less than the second writing event sequence number.
[0013] In some embodiments, the audio data includes a first audio segment and a second audio segment that occur sequentially; configuring an audio event sequence number for the corresponding audio data using a counter includes: A counter is used to configure the first audio event sequence number for the first audio segment; A counter is used to configure the second audio event sequence number for the second audio segment; The first audio event sequence number is less than the second audio event sequence number.
[0014] In some embodiments, the first writing event sequence number, the second writing event sequence number, the first audio event sequence number, and the second audio event sequence number exhibit a monotonically increasing trend according to the chronological order in which the first writing segment, the second writing segment, the first audio segment, and the second audio segment occur.
[0015] In some embodiments, establishing a mapping relationship between the writing event identifier and the audio event identifier, and storing the writing data and the corresponding audio data, includes: Establish and store the correspondence between the writing event sequence number and the audio event sequence number; The writing event sequence number and the writing data are stored together. The audio event sequence number and the corresponding audio data are stored together.
[0016] In some embodiments, when there are multiple audio event sequence numbers, the associated storage of the writing event sequence number, the audio event sequence number, and the corresponding audio data includes: Determine the first audio event sequence number that follows the written event sequence number among the plurality of audio event sequence numbers; The writing event sequence number, the first audio event sequence number, and the corresponding audio data are stored together.
[0017] In some embodiments, the multimodal conference data synchronization method further includes: generating a conference identifier in response to a user's operation of opening a conference record; the conference identifier is used for associated storage with the multimodal conference data.
[0018] In some embodiments, the multimodal conference data synchronization method further includes: recording the note page number corresponding to the writing event; the note page number is used for associated storage with the multimodal conference data.
[0019] In some embodiments, the multimodal conference data synchronization method further includes: Based on the writing content clicked by the user, determine the target writing data corresponding to the writing content; The target audio data is determined based on the writing event identifier corresponding to the target writing data and the mapping relationship; Play the target audio data.
[0020] In some embodiments, determining the target audio data based on the writing event identifier corresponding to the target writing data and the mapping relationship includes: Based on the target writing data, query the associated storage of the writing event sequence number and the writing data to determine the writing event sequence number corresponding to the target writing data; The corresponding target audio event sequence number is determined by querying the mapping relationship based on the writing event sequence number corresponding to the target writing data; The target audio data is obtained by querying the associated stored audio event sequence number and the corresponding audio data based on the target audio event sequence number.
[0021] Secondly, this application also provides a multimodal conference data synchronization system, which includes: The writing event detection module is configured to detect writing events and configure writing event identifiers for the corresponding writing data. The audio event detection module is configured to detect an audio event when the writing event is detected, collect the corresponding audio data, and configure an audio event identifier for the corresponding audio data. The data storage module is configured to establish a mapping relationship between the writing event identifier and the audio event identifier, and to store the writing data and the corresponding audio data.
[0022] In some embodiments, the multimodal conference data synchronization system further includes: The writing data determination module is configured to determine the target writing data corresponding to the writing content clicked by the user. The audio data determination module is configured to determine the target audio data based on the writing event identifier corresponding to the target writing data and the mapping relationship; An audio data playback module is configured to play the target audio data.
[0023] Thirdly, this application also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of any of the multimodal conference data synchronization methods of this application.
[0024] Fourthly, this application also provides a computer-readable storage medium storing a computer program / instructions thereon, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the multimodal conference data synchronization method described in any one of this application.
[0025] Fifthly, this application also provides a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the multimodal conference data synchronization method described in any one of this application.
[0026] The multimodal conference data synchronization method of this application achieves synchronization between written data and audio data during the conference by configuring associated written event identifiers and audio event identifiers for written data and audio data, establishing a mapping relationship between the written event identifiers and audio event identifiers, and storing the written data and corresponding audio data. The synchronization process does not require complex and precise timestamps, avoiding dependence on the accuracy and stability of the RTC. Attached Figure Description
[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0028] Figure 1 A flowchart illustrating an embodiment of the multimodal conference data synchronization method of this application; Figure 2 A flowchart of another embodiment of the multimodal conference data synchronization method of this application; Figure 3 A flowchart of another embodiment of the multimodal conference data synchronization method of this application; Figure 4A flowchart of another embodiment of the multimodal conference data synchronization method of this application; Figure 5 This is a schematic block diagram of an embodiment of the multimodal conference data synchronization system of this application; Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation
[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.
[0030] It should also be noted that, in this document, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0031] like Figure 1 As shown, an embodiment of this application provides a multimodal conference data synchronization method, which includes: S10. Detect writing events and configure writing event identifiers for the corresponding writing data.
[0032] Exemplarily, the above steps can be performed using devices such as smart pens, notebooks, and tablets, where the notebooks and tablets are equipped with electromagnetic pens or styluses. Users can use a smart pen to write on dot-matrix paper, an electromagnetic pen to write on a notebook, or a stylus to write on a tablet, i.e., take notes during meetings. In some embodiments, the smart pen can also be used with an external terminal. This external terminal can be a smartphone, tablet computer, laptop computer, notebook computer, etc., and this application is not limited thereto. The following embodiments only use the use of a smart pen with a smartphone as an example.
[0033] A writing event can be, for example, a writing process from pen down to pen up (e.g., each stroke of the pen is a writing event, or circles, strikethroughs, underlines, question marks, exclamation marks, asterisks, arrows, etc., drawn by the user while taking notes are each writing events). When a writing event is detected, a writing event identifier is configured for that writing event.
[0034] In some embodiments, the multimodal conference data synchronization method further includes: recording the note page number corresponding to the writing event; the note page number is used for associated storage with the multimodal conference data, and the note page number can improve the data retrieval efficiency during playback.
[0035] For example, writing data includes, but is not limited to, pen tip trajectory data (e.g., coordinate values), writing pressure values, and note page numbers (page_id). The trajectory data can be obtained through a dot matrix recognition module integrated into the smart pen (or notebook or tablet), for example, identifying the coordinates (x, y) of the pen tip on dot matrix paper; the writing pressure value can be detected by a pressure sensor integrated into the smart pen (or notebook or tablet), and the writing pressure value corresponds to the coordinates; the note page number is, for example, the page number of the page the user is currently taking notes on (e.g., the page number of a PDF file the user is viewing through an external terminal during a meeting).
[0036] S20. When the writing event is detected, an audio event is detected, and corresponding audio data is collected and an audio event identifier is configured for the corresponding audio data.
[0037] For example, a smart pen can perform both writing and audio recording, as well as audio event detection; a notebook can be used to write with an electromagnetic pen and record audio and detect audio events using its built-in microphone; a tablet / pad can be used to write with a stylus and record audio and detect audio events using its built-in microphone. When a writing event is detected, audio event detection is triggered to detect the audio event. For example, audio events can be detected using VAD (Voice Activity Detection) technology, and corresponding audio data can be collected and configured with audio event identifiers for the corresponding audio data. Audio data includes, but is not limited to, audio segments and the duration of audio segments.
[0038] For example, audio capture can be automatically initiated when a writing event is detected, or the user can manually trigger the capture of audio segments (e.g., one audio segment every 500ms).
[0039] S30. Establish a mapping relationship between the writing event identifier and the audio event identifier, and store the writing data and the corresponding audio data.
[0040] For example, a correspondence is established and stored between the writing event sequence number and the audio event sequence number, for instance, as a mapping table {seq_id_range, data_type, file_pointer}, where seq_id_range is the sequence number range, data_type is the data type, and file_pointer is the file pointer. The writing event sequence number and the writing data are also stored in association; for example, the writing event identifier and the writing data are packaged and stored as {seq_id, x, y, pressure, page_id}, where seq_id is the writing event sequence number. Similarly, the audio event sequence number and the corresponding audio data are stored in association; for example, the audio event identifier and the corresponding audio data are packaged and stored as {seq_id, audio_segment, duration}, where seq_id is the audio event sequence number, audio_segment is the audio segment, and duration is the duration of the audio segment. This allows the corresponding audio data to be retrieved based on the pre-established mapping relationship when accessing meeting data subsequently. The writing data and the corresponding audio data can be stored on local storage media or synchronized to the cloud; this application does not limit this.
[0041] The multimodal conference data synchronization method of this application achieves synchronization between written data and audio data during the conference by configuring associated written event identifiers and audio event identifiers for written data and audio data, establishing a mapping relationship between the written event identifiers and audio event identifiers, and storing the written data and corresponding audio data. The synchronization process does not require complex and precise timestamps, avoiding dependence on the accuracy and stability of the RTC.
[0042] like Figure 2 The diagram shown is a flowchart of another embodiment of the multimodal conference data synchronization method of this application. In this embodiment, the writing event identifier is a writing event sequence number, and the audio event identifier is an audio event sequence number. The writing event sequence number and the audio event sequence number exhibit a monotonically increasing trend according to the chronological order of the writing data and the corresponding audio data. The multimodal conference data synchronization method further includes: S40. A counter is used to configure a writing event sequence number for the writing data. The writing data includes at least one writing segment.
[0043] For example, the writing data includes a first writing segment and a second writing segment that occur sequentially; configuring a writing event sequence number for the writing data using a counter includes: configuring a first writing event sequence number for the first writing segment using the counter; configuring a second writing event sequence number for the second writing segment using the counter; the first writing event sequence number is less than the second writing event sequence number.
[0044] S50. Configure an audio event sequence number for the corresponding audio data using a counter. The corresponding audio data includes at least one audio segment.
[0045] For example, the audio data includes a first audio segment and a second audio segment that occur sequentially; configuring an audio event sequence number for the corresponding audio data using a counter includes: configuring a first audio event sequence number for the first audio segment using a counter; configuring a second audio event sequence number for the second audio segment using a counter; the first audio event sequence number is less than the second audio event sequence number.
[0046] In some embodiments, the first writing event sequence number, the second writing event sequence number, the first audio event sequence number, and the second audio event sequence number exhibit a monotonically increasing trend according to the chronological order in which the first writing segment, the second writing segment, the first audio segment, and the second audio segment occur.
[0047] For example, the counter can be a dedicated counter chip or an internal counter of an FPGA. The counter frequency is 1kHz~10kHz (adjustable according to accuracy requirements), and the counter bit width is 32 bits or 64 bits, supporting no overflow during the entire meeting recording process. Reset condition: It can be reset at the start of a new meeting. In this embodiment, the acquisition of written data and audio data is divided into atomic operations. An atomic operation is one or a series of operations that cannot be interrupted. Either all operations are executed successfully, or none are executed at all, and there will be no situation where "execution is interrupted halfway". Combined with the counter, the atomic operation increment = read the counter value → increment by 1 → write back to the counter. These three steps are executed as an indivisible whole and will not be interrupted by other events, thereby ensuring that the sequence number obtained by each event is unique.
[0048] In some embodiments, there may be a one-to-one correspondence between written segments and audio segments, or one written segment may correspond to two or more audio segments, or two or more written segments may correspond to one audio segment.
[0049] For example, the order in which the first writing segment, the second writing segment, the first audio segment, and the second audio segment occur can be: first writing segment, first audio segment, second writing segment, and second audio segment. The corresponding sequence numbers for the first writing event, the first audio event, the second writing event, and the second audio event can be X, X+1, X+2, and X+3, respectively, where X is the initial sequence number. Each writing segment corresponds to one audio segment, meaning there is a one-to-one correspondence between writing segments and audio segments.
[0050] For example, the order in which the first writing segment, the second writing segment, the first audio segment, and the second audio segment occur can be: first writing segment, first audio segment, second audio segment, and second writing segment. The corresponding sequence numbers for the first writing event, the first audio event, the second audio event, and the second writing event can be X, X+1, X+2, and X+3, respectively, where X is the initial sequence number. The first writing segment corresponds to two audio segments (the first audio segment and the second audio segment), meaning one writing segment corresponds to two or more audio segments.
[0051] For example, the order in which the first writing segment, the second writing segment, and the first audio segment occur can be: the first writing segment, the second writing segment, and the first audio segment. The corresponding sequence numbers of the first writing event, the second writing event, and the first audio event can be X, X+1, and X+2, respectively, where X is the initial sequence number. Two writing segments (the first writing segment and the second writing segment) correspond to one audio segment; that is, two or more writing segments correspond to one audio segment.
[0052] The above three situations can be further illustrated in Table 1 below: Table 1
[0053] like Figure 3 The diagram shows a flowchart of another embodiment of the multimodal conference data synchronization method of this application. In this embodiment, when there are multiple audio event sequence numbers, the associated storage of the writing event sequence number, the audio event sequence number, and the corresponding audio data includes: S31. Determine the first audio event sequence number that is located after the written event sequence number among the plurality of audio event sequence numbers.
[0054] For example, given a written event sequence number seq_id: 10, and audio event sequence numbers seq_id: 11, 12, 13, 14, 15, 16, then audio event sequence number seq_id=11 is the first audio event sequence number after written event sequence number seq_id=10. Similarly, given written event sequence numbers seq_id: 20, 21, 22, and audio event sequence numbers seq_id: 23, 24, then audio event sequence number seq_id=23 is the first audio event sequence number after written event sequence number seq_id=20.
[0055] S32. The writing event sequence number, the first audio event sequence number, and the corresponding audio data are associated and stored.
[0056] In this embodiment, the association between written data and audio data is established through the numerical range of seq_id, rather than timestamps. For example, the audio segment corresponding to the written event sequence number seq_id=10 is the closest audio event seq_id=11 with a seq_id greater than 10.
[0057] In some embodiments, the multimodal conference data synchronization method further includes: generating a conference identifier in response to a user's operation of opening a conference record; the conference identifier is used for associated storage with the multimodal conference data.
[0058] For example, a user initiates meeting recording by pressing the start recording button. Upon detecting this action, the smart pen, notebook, or tablet automatically generates a meeting identifier (e.g., a meeting event ID) and associates this meeting event ID with the multimodal meeting data for distinguishing multimodal meeting data from different meetings. For example, all data from a single meeting (written files + audio files + mapping table) is associated with the same folder, such as Session_S001 / .
[0059] In some embodiments, the multimodal conference data synchronization method further includes recording the RTC time of the session start and / or the RTC time of the session end. For example, the "Meeting Record of 2026-04-02 14:00" is displayed in the stored file list, allowing users to easily search for historical records by actual time. It should be noted that the RTC time here is only used to record the start and / or end times of the meeting and is not used for alignment between multimodal conference data.
[0060] In this embodiment, the meeting event ID is used to distinguish the written data and audio data of different meetings, which are multimodal meeting data. This can achieve spatial independence between the sequence numbers of different sessions (written event sequence number and audio event sequence number), which facilitates the management of multimodal data of different meetings.
[0061] For example, a session event: • Meaning: Marks the boundary of a complete recording process, from when the user clicks "Start Recording" to when they click "End Recording"; • Data content: {session_id: “S001”, start_seq_id: 1, end_seq_id: 5000,rtc_time: “2026-04-02 14:00”}; A session event can contain multiple audio events and multiple writing events.
[0062] • seq_id range: The start and end range of seq_id in this session (e.g., 1-5000) for the session event log. For example: Session events: seq_id range 1-10000 ├── Writing events: seq_id = 1, 5, 8, 12... (corresponding to actual writing times) ├── Audio events: seq_id = 2, 3, 4, 6, 7... (one audio segment every 500ms) └── Browsing event: seq_id = 100 (page turning action).
[0063] Among them, browsing events correspond to digital content browsing data, which refers to a third type of data stream besides writing and audio during the user's recording session, for example: • Dot matrix paper page switching: The user's action of turning the page from page 1 to page 2; • Electronic document viewing: Scrolling and page-turning operations when viewing PDFs / images in the accompanying APP; • Hyperlink Click: Click on printed links on the dot matrix paper (such as URLs in the meeting agenda).
[0064] For example, during meeting minutes, if a user views page 3 of the meeting materials PDF, this browsing action will generate a "Digital Content Browsing Data" entry containing: {Page ID: 3, Browsing Start seq_id: 150, Browsing End seq_id: 280}.
[0065] In some embodiments, a meeting typically corresponds to one session event, but it is also possible for a meeting to be recorded as two session events through user interaction, as shown in Table 2 below: Table 2
[0066] like Figure 4 The following is a flowchart showing another embodiment of the multi-modal conference data synchronization method of the present application. In this embodiment, in some embodiments, the multi-modal conference data synchronization method further includes: S60. Determine the target writing data corresponding to the written content according to the written content clicked by the user.
[0067] Exemplarily, when the user views the meeting minutes after the meeting, the user can click on the notes made during the meeting (i.e., the written content, for example, the words written by the user or the circles, strikethroughs, underlines, question marks, exclamation marks, asterisks, arrows, etc. drawn by the user). After detecting the written content clicked by the user, further determine the corresponding target writing data according to the written content (for example, the trajectory data of the pen tip and / or the writing pressure value and / or the note page number, etc.).
[0068] S70. Determine the target audio data according to the writing event identifier corresponding to the target writing data and the mapping relationship.
[0069] Exemplarily, query the associated stored writing event serial number and the writing data according to the target writing data to determine the writing event serial number corresponding to the target writing data; query the mapping relationship according to the writing event serial number corresponding to the target writing data to determine the corresponding target audio event serial number; query the associated stored audio event serial number and the corresponding audio data according to the target audio event serial number to obtain the corresponding target audio data.
[0070] Exemplarily, after determining the target writing data, query the data packet of the previously stored writing data to find the writing event identifier (for example, the writing event serial number) corresponding to the target writing data; further combine the pre-established mapping relationship to determine the corresponding target audio event identifier (for example, the target audio event serial number); finally, retrieve the corresponding target audio data according to the target audio event serial number.
[0071] S80. Play the target audio data.
[0072] Exemplarily, call the playback module (for example, the speaker of the smart pen or the speaker of the office notebook or the speaker of the Pad / tablet) to play the target audio data.
[0073] In some embodiments, taking the example that the user writes 5 strokes (3 characters of "Hello everyone") and records audio for 2 seconds (the audio is segmented every 500 ms) in a meeting, the multi-modal conference data synchronization method is described as follows: Assume that each audio segment is 500 ms, and the writing events occur as they actually do. The specific multi-modal conference data recorded is shown in Table 3 below: Table 3
[0074] When recording the multimodal conference data in this embodiment for different event types, the following event granularities can be referred to for data recording. See Table 4 below for details:
[0075] Exemplarily, when the user writes the character 'one' (which takes 500 ms and generates 50 coordinate points), using the fine granularity: 50 writing events are generated, and the seq_ids are 100, 101, 102... 149 respectively (one seq_id for each point). In this embodiment, during playback, it can be accurate to the audio content corresponding to 'the 3rd millisecond of writing the horizontal line'.
[0076] The data correspondence in Table 3 above is as follows: • 5 writing event sequence numbers: seq_id = 1, 2, 3, 4, 5 (the character 'big'); • 4 audio event sequence numbers: seq_id = 6, 12, 18, 24 (one segment every 500 ms); The seq_ids of the writing data and audio data in the above embodiments are interleaved and monotonically increasing in chronological order. The corresponding relationship is established by 'finding the closest greater sign'.
[0077] After obtaining the above writing events and audio events, the corresponding writing data, audio data, and event mapping table are encapsulated in the following manner: Writing data packet format: struct HandwritingPacket { uint32_t seq_id; / / Event sequence number uint16_t x; / / X coordinate uint16_t y; / / Y coordinate uint8_t pressure; / / Pressure value uint16_t page_id; / / Page ID (dot matrix paper recognition) / / Note: Does not include the timestamp field }; Audio data packet format: struct AudioPacket { uint32_t seq_id; / / Event sequence number (associated with the writing data) uint16_t duration_ms; / / Audio segment duration (used for playback) uint32_t data_offset; / / Offset within the audio file uint32_t data_size; / / Audio data size / / Note: Does not include a timestamp field; does not record absolute time. }; "Offset within audio file" refers to the byte position of the audio segment within the complete audio file.
[0078] Specific examples: • A meeting recorded 10 minutes of audio, which was saved as meeting.wav (total size: 10MB = 10,485,760 bytes). • The audio is segmented in 500ms segments, each segment being 40,960 bytes; • Information from the third audio segment (seq_id=6); • seq_id: 6 • data_offset: 81920 (Skip the first two segments, the third segment starts from byte 81920); • data_size: 40960 (This segment occupies 40960 bytes); • duration_ms: 500 Function: During playback, it can directly jump to the 81920th byte of the file based on the data_offset to read the data, without scanning the entire file, achieving millisecond-level positioning.
[0079] "Audio data size" refers to the number of bytes per audio segment. Calculation example: • Audio format: PCM, 16bit, mono, 16kHz sampling rate; • Duration of each segment: 500ms; • Data size calculation: 16000Hz × 0.5s × 2 bytes (16bit) × 1 channel = 16,000 bytes; Therefore, the data_size field value for each audio event is approximately 16000 (the actual value may be slightly higher, including header information).
[0080] Event Mapping Table (Core Innovation): struct EventMapping { uint32_t seq_id_start; / / Start of sequence number uint32_t seq_id_end; / / End of sequence number uint8_t data_type; / / Data type (writing / audio / other) uint32_t file_pointer; / / File position pointer / / Note: Timestamp field is not included. }
[0081] The process for users to view the meeting minutes after the meeting is as follows: 1. The user clicks on a point on the writing path; 2. The system obtains the event sequence number (seq_id_target) of the stroke to which the point belongs; 3. Query the event mapping table to find the audio data packet containing seq_id_target; 4. Playback begins from this audio data packet and continues sequentially according to the sequence number.
[0082] For example: When a user clicks the word "big" (seq_id=3), the system searches for the closest audio seq_id=6 with a value greater than 3, and plays audio segment #1 (0-500ms); when a user clicks the word "home" (seq_id=7), the system searches for the closest audio seq_id=12 with a value greater than 7, and plays audio segment #2 (500-1000ms).
[0083] The multimodal meeting data synchronization method of this application can be applied in various fields. When used in the smart pen education market, it can synchronize student classroom notes with teacher explanations, supporting the playback of corresponding audio by clicking on notes. When used for business meeting recording, it can link meeting minutes with discussion audio for easy review and confirmation later. When used for news interviews, it can synchronize interview recordings with audio recordings, quickly locating key information. When used for medical records, it can synchronize doctors' handwritten medical records with oral records, ensuring the integrity of the records. When used for legal evidence collection, it can synchronize handwritten transcripts with on-site audio recordings, enhancing the credibility of evidence. When used for industrial design, it can synchronize hand-drawn sketches with design discussion audio, preserving design ideas.
[0084] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0085] like Figure 5 The diagram shown is a schematic block diagram of an embodiment of the multimodal conference data synchronization system of this application. In this embodiment, the multimodal conference data synchronization system includes: The writing event detection module 510 is configured to detect writing events and configure writing event identifiers for the corresponding writing data. The audio event detection module 520 is configured to detect an audio event when the writing event is detected, collect corresponding audio data, and configure an audio event identifier for the corresponding audio data. The data storage module 530 is configured to establish a mapping relationship between the writing event identifier and the audio event identifier, and to store the writing data and the corresponding audio data.
[0086] In some embodiments, the multimodal conference data synchronization system further includes: The writing data determination module is configured to determine the target writing data corresponding to the writing content clicked by the user. The audio data determination module is configured to determine the target audio data based on the writing event identifier corresponding to the target writing data and the mapping relationship; An audio data playback module is configured to play the target audio data.
[0087] In some embodiments, the writing event identifier is a writing event sequence number, and the audio event identifier is an audio event sequence number; the writing event sequence number and the audio event sequence number exhibit a monotonically increasing trend according to the chronological order in which the writing data and the corresponding audio data occur; the method further includes: A counter is used to configure the writing event sequence number for the written data; A counter is used to configure the audio event sequence number for the corresponding audio data.
[0088] In some embodiments, the writing data includes at least one writing segment, and the corresponding audio data includes at least one audio segment.
[0089] In some embodiments, the writing data includes a first writing segment and a second writing segment that occur sequentially; configuring the writing event sequence number for the writing data using a counter includes: The first writing event sequence number is configured for the first writing segment using the counter; The counter is used to configure a second writing event sequence number for the second writing segment; The first writing event sequence number is less than the second writing event sequence number.
[0090] In some embodiments, the audio data includes a first audio segment and a second audio segment that occur sequentially; configuring an audio event sequence number for the corresponding audio data using a counter includes: A counter is used to configure the first audio event sequence number for the first audio segment; A counter is used to configure the second audio event sequence number for the second audio segment; The first audio event sequence number is less than the second audio event sequence number.
[0091] In some embodiments, the first writing event sequence number, the second writing event sequence number, the first audio event sequence number, and the second audio event sequence number exhibit a monotonically increasing trend according to the chronological order in which the first writing segment, the second writing segment, the first audio segment, and the second audio segment occur.
[0092] In some embodiments, establishing a mapping relationship between the writing event identifier and the audio event identifier, and storing the writing data and the corresponding audio data, includes: The writing event sequence number and the writing data are stored together. The writing event sequence number, the audio event sequence number, and the corresponding audio data are stored together.
[0093] In some embodiments, when there are multiple audio event sequence numbers, the associated storage of the writing event sequence number, the audio event sequence number, and the corresponding audio data includes: Determine the first audio event sequence number that follows the written event sequence number among the plurality of audio event sequence numbers; The writing event sequence number, the first audio event sequence number, and the corresponding audio data are stored together.
[0094] In some embodiments, the method further includes: generating a meeting identifier in response to a user's operation of starting a meeting record; the meeting identifier is used for associated storage with the multimodal meeting data.
[0095] In some embodiments, the method further includes: recording the note page number corresponding to the writing event; the note page number is used for associated storage with the multimodal conference data.
[0096] The multimodal conference data synchronization system described in the above embodiments of this application can be used to execute the multimodal conference data synchronization method described in the above embodiments of this application, and accordingly achieve the technical effects achieved by the multimodal conference data synchronization method described in the above embodiments of this application, which will not be elaborated here. In the embodiments of this application, the relevant functional modules can be implemented using a hardware processor.
[0097] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the multimodal conference data synchronization methods described above.
[0098] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described multimodal conference data synchronization methods.
[0099] In some embodiments, this application also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a multimodal conference data synchronization method.
[0100] Figure 6 This is a schematic diagram of the hardware structure of an electronic device for performing a multimodal conference data synchronization method according to another embodiment of this application, as shown below. Figure 6 As shown, the device includes: One or more processors 610 and memory 620, Figure 6 Take the 610 processor as an example.
[0101] The device for performing the multimodal conference data synchronization method may further include an input device 630 and an output device 640.
[0102] The processor 610, memory 620, input device 630, and output device 640 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.
[0103] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the multimodal conference data synchronization method in the embodiments of this application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the multimodal conference data synchronization method in the above-described method embodiments.
[0104] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the multimodal conference data synchronization device. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories can be connected to the multimodal conference data synchronization device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0105] Input device 630 can receive input digital or character information and generate signals related to user settings and function control of the multimodal conference data synchronization device. Output device 640 may include display devices such as a display screen.
[0106] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, they execute the multimodal conference data synchronization method in any of the above method embodiments.
[0107] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.
[0108] The electronic devices in this application embodiments exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.
[0109] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include: laptops, PDAs, MIDs, and UMPC devices, such as the iPad.
[0110] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.
[0111] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.
[0112] (5) Other electronic devices with data interaction functions.
[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for synchronizing multimodal conference data, characterized in that, include: Detect writing events and configure writing event identifiers for the corresponding writing data; When the writing event is detected, an audio event is detected, and corresponding audio data is collected and an audio event identifier is configured for the corresponding audio data; Establish a mapping relationship between the writing event identifier and the audio event identifier, and store the writing data and the corresponding audio data.
2. The method according to claim 1, characterized in that, The writing event identifier uses a writing event sequence number, and the audio event identifier uses an audio event sequence number. The writing event sequence number and the audio event sequence number show a monotonically increasing trend according to the chronological order in which the writing data and the corresponding audio data occur. The method further includes: A counter is used to configure the writing event sequence number for the written data; A counter is used to configure the audio event sequence number for the corresponding audio data.
3. The method according to claim 2, characterized in that, The written data includes at least one written segment, and the corresponding audio data includes at least one audio segment.
4. The method according to claim 3, characterized in that, The counter is configured with the writing event sequence number and / or the audio event sequence number for the written data and / or the corresponding audio data using an atomic incrementing method.
5. The method according to claim 3, characterized in that, The written data includes a first written segment and a second written segment that occurred sequentially. The step of configuring the writing event sequence number for the writing data using a counter includes: The first writing event sequence number is configured for the first writing segment using the counter; The counter is used to configure a second writing event sequence number for the second writing segment; The first writing event sequence number is less than the second writing event sequence number.
6. The method according to claim 5, characterized in that, The audio data includes a first audio segment and a second audio segment that occur sequentially; configuring the audio event sequence number for the corresponding audio data using a counter includes: A counter is used to configure the first audio event sequence number for the first audio segment; A counter is used to configure the second audio event sequence number for the second audio segment; The first audio event sequence number is less than the second audio event sequence number.
7. The method according to claim 6, characterized in that, The first writing event sequence number, the second writing event sequence number, the first audio event sequence number, and the second audio event sequence number exhibit a monotonically increasing trend according to the chronological order in which the first writing segment, the second writing segment, the first audio segment, and the second audio segment occur.
8. The method according to claim 7, characterized in that, The step of establishing a mapping relationship between the writing event identifier and the audio event identifier, and storing the writing data and the corresponding audio data, includes: Establish and store the correspondence between the writing event sequence number and the audio event sequence number; The writing event sequence number and the writing data are stored together. The audio event sequence number and the corresponding audio data are stored together.
9. The method according to claim 8, characterized in that, When there are multiple audio event sequence numbers, the associated storage of the writing event sequence number, the audio event sequence number, and the corresponding audio data includes: Determine the first audio event sequence number that follows the written event sequence number among the plurality of audio event sequence numbers; The writing event sequence number, the first audio event sequence number, and the corresponding audio data are stored together.
10. The method according to any one of claims 1-9, characterized in that, Also includes: In response to a user's action of starting a meeting record, a meeting identifier is generated; the meeting identifier is used to associate and store the multimodal meeting data.
11. The method according to claim 10, characterized in that, Also includes: Record the note page number corresponding to the writing event; the note page number is used for associated storage with the multimodal meeting data.
12. The method according to any one of claims 1-9, characterized in that, Also includes: Based on the writing content clicked by the user, determine the target writing data corresponding to the writing content; The target audio data is determined based on the writing event identifier corresponding to the target writing data and the mapping relationship; Play the target audio data.
13. The method according to claim 12, characterized in that, The step of determining the target audio data based on the writing event identifier corresponding to the target writing data and the mapping relationship includes: Based on the target writing data, query the associated storage of the writing event sequence number and the writing data to determine the writing event sequence number corresponding to the target writing data; The corresponding target audio event sequence number is determined by querying the mapping relationship based on the writing event sequence number corresponding to the target writing data; The target audio data is obtained by querying the associated stored audio event sequence number and the corresponding audio data based on the target audio event sequence number.
14. A multimodal conference data synchronization system, characterized in that, include: The writing event detection module is configured to detect writing events and configure writing event identifiers for the corresponding writing data. The audio event detection module is configured to detect an audio event when the writing event is detected, collect the corresponding audio data, and configure an audio event identifier for the corresponding audio data. The data storage module is configured to establish a mapping relationship between the writing event identifier and the audio event identifier, and to store the writing data and the corresponding audio data.
15. The system according to claim 14, characterized in that, Also includes: The writing data determination module is configured to determine the target writing data corresponding to the writing content clicked by the user. The audio data determination module is configured to determine the target audio data based on the writing event identifier corresponding to the target writing data and the mapping relationship; An audio data playback module is configured to play the target audio data.
16. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-13.
17. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-13.
18. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-13.