Data processing method and device, computer device, readable storage medium and program product

Abstract

The application relates to a data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: obtaining recording data and handwriting data of a handwriting pen, determining text data corresponding to the recording data, and determining third time stamps corresponding to each character in the text data according to each first time stamp; and fusing multiple frames of voice data and multiple frames of handwriting image based on the first time stamp, the second time stamp and the third time stamp to obtain target fusion data. The method can improve the fusion effect of the recording data and the handwriting data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the continuous development of electronic technology and the continuous improvement of users' living standards, various electronic devices have gradually become popular in users' lives. Among them, styluses have gradually become one of the commonly used electronic devices in users' lives due to their advantages such as convenience, ease of use, ability to reproduce handwriting, and strong adaptability.

[0003] Currently, styluses can also record audio. In some application scenarios, users can use styluses to record audio while writing handwriting. However, traditional technologies suffer from poor data fusion between handwriting and audio recordings. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the data fusion effect of handwriting and audio recording, in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a data processing method, comprising: acquiring audio recording data and handwriting data from a stylus, wherein the audio recording data includes multiple frames of speech data, each speech data including a first timestamp, and the handwriting data includes multiple frames of handwriting images, each handwriting image including a second timestamp; determining text data corresponding to the audio recording data, and determining a third timestamp corresponding to each character in the text data based on each first timestamp; and fusing the multiple frames of speech data and the multiple frames of handwriting images based on the first timestamp, the second timestamp, and the third timestamp to obtain target fused data.

[0006] In one embodiment, acquiring audio recording data and handwriting data from a stylus includes: receiving audio recording data and handwriting data sent by the stylus; wherein the audio recording data and handwriting data are obtained by the stylus responding to a recording command by timestamping each frame of handwriting image and each frame of audio data based on a first timestamp and a second timestamp corresponding to the current time.

[0007] In one embodiment, acquiring audio recording data and handwriting data from a stylus includes: in response to a recording command, acquiring audio recording data through an audio acquisition component and sending a handwriting acquisition command to the stylus; in response to a time calibration request sent by the stylus, sending a first timestamp and / or a second timestamp corresponding to the current time to the stylus; and in response to a recording end command, receiving handwriting data sent by the stylus, and saving the handwriting data and audio recording data if each first timestamp matches each second timestamp.

[0008] In one embodiment, multiple frames of voice data and multiple frames of handwriting images are fused based on a first timestamp, a second timestamp, and a third timestamp to obtain target fused data. This includes: for each second timestamp, traversing the target third timestamp that matches the second timestamp in each third timestamp, determining the target first timestamp that matches the target third timestamp in each first timestamp, and combining the voice data corresponding to the target first timestamp and the handwriting image corresponding to the second timestamp to obtain a data group; and fusing the data groups to obtain the target fused data.

[0009] In one embodiment, fusing the data groups to obtain target fused data includes: determining the index marker of each data group based on the temporal order of each second timestamp in the handwriting data; and fusing the data groups according to each index marker to obtain target fused data.

[0010] In one embodiment, the method further includes: in response to a trigger command for a first handwriting image in the target fused data, querying a target index marker corresponding to the first handwriting image, determining a first timestamp corresponding to the target index marker, and jumping to play the first voice data corresponding to the first timestamp of the target index marker.

[0011] In one embodiment, the method further includes: in response to a trigger command for first speech data in the target fused data, querying a target index marker corresponding to the first speech data, determining a second timestamp corresponding to the target index marker, and displaying a first handwriting image of the second timestamp corresponding to the target index marker.

[0012] Secondly, this application also provides a data processing apparatus, comprising: a data receiving module for acquiring audio recording data and handwriting data from a stylus, wherein the audio recording data includes multiple frames of audio data, each audio data including a first timestamp, and the handwriting data includes multiple frames of handwriting images, each handwriting image including a second timestamp; a timing module for determining text data corresponding to the audio recording data and determining a third timestamp corresponding to each character in the text data based on each first timestamp; and a data fusion module for fusing the multiple frames of audio data and the multiple frames of handwriting images based on the first timestamp, the second timestamp, and the third timestamp to obtain target fused data.

[0013] Thirdly, this application provides another data processing method, including: acquiring target data, the target data including audio recording data and / or handwriting data, the audio recording data including multiple frames of audio data, each audio data including a first timestamp, the handwriting data including multiple frames of handwriting images, each handwriting image including a second timestamp; sending the target data to a terminal, the target data being used by the terminal to fuse the multiple frames of audio data and the multiple frames of handwriting images based on the first timestamp, the second timestamp, and a third timestamp to obtain target fused data, the third timestamp being the timestamp corresponding to each character in the text data corresponding to the audio recording data.

[0014] In one embodiment, acquiring target data includes: in response to a recording command, acquiring multiple frames of candidate handwriting images through an image acquisition component and acquiring multiple frames of candidate speech data through an audio acquisition component; for each frame of candidate handwriting image and each frame of candidate speech data, timestamping the candidate speech data based on a first timestamp corresponding to the current time to obtain speech data, and timestamping the candidate handwriting image based on a second timestamp corresponding to the current time to obtain handwriting image; in response to a recording end command, combining each speech data in chronological order to obtain recording data, and combining each handwriting image in chronological order to obtain handwriting data.

[0015] Fourthly, this application also provides another data processing apparatus, comprising: a data acquisition module for acquiring target data, the target data including audio recording data and / or handwriting data, the audio recording data including multiple frames of audio data, each audio data including a first timestamp, the handwriting data including multiple frames of handwriting images, each handwriting image including a second timestamp; and a data transmission module for transmitting the target data to a terminal, the target data being used by the terminal to fuse the multiple frames of audio data and the multiple frames of handwriting images based on the first timestamp, the second timestamp, and a third timestamp to obtain target fused data, the third timestamp being the timestamp corresponding to each character in the text data corresponding to the audio recording data.

[0016] Fifthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method provided in the first aspect or the third method described above.

[0017] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods provided in the first or third aspect above.

[0018] In a seventh aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods provided in the first or third aspect above.

[0019] The aforementioned data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product collect audio data and / or handwriting data using a stylus and send the audio data and / or handwriting data to a terminal. The audio data includes multiple frames of audio data, each with a first timestamp. The handwriting data includes multiple frames of handwriting images, each with a second timestamp. The terminal performs text-to-text processing on the audio data to obtain corresponding text data, and determines a third timestamp for each character in the text data based on the first timestamp. Based on the first, second, and third timestamps, the multiple frames of audio data and multiple frames of handwriting images are fused to obtain target fused data. Through the combined operation of the stylus and the terminal, the timestamps of the audio data and handwriting data are aligned, ensuring temporal consistency between the audio data and handwriting data. This allows for synchronization of handwriting and audio recordings when the target fused data is displayed, improving the fusion effect of the audio data and handwriting data. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0021] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.

[0022] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0023] Figure 3 This is a flowchart illustrating step 201 in one embodiment;

[0024] Figure 4 This is a flowchart illustrating step 201 in another embodiment;

[0025] Figure 5 This is a flowchart illustrating step 203 in one embodiment;

[0026] Figure 6 This is a flowchart illustrating step 502 in one embodiment;

[0027] Figure 7 This is a flowchart illustrating the target fusion data in one embodiment;

[0028] Figure 8 This is a flowchart illustrating the target fusion data in another embodiment;

[0029] Figure 9 This is a flowchart illustrating the data processing method in another embodiment;

[0030] Figure 10 This is a flowchart illustrating another data processing method in one embodiment;

[0031] Figure 11 This is a flowchart illustrating step 1001 in another embodiment;

[0032] Figure 12 This is a structural block diagram of a data processing device in one embodiment;

[0033] Figure 13 A structural block diagram of another data processing apparatus in one embodiment; and

[0034] Figure 14 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0036] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0037] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown includes at least a stylus 101 and a terminal 102.

[0038] The stylus 101 is used to collect handwriting data and audio data in response to a recording command submitted by the user, or to collect handwriting data and send the handwriting data, or the handwriting data and audio data, to the terminal 102 in response to a handwriting collection command sent by the terminal 102. The stylus 101 can be a capacitive pen, a smart pen, a stylus with a data processing unit, a ballpoint pen with a data processing unit, a fountain pen with a data processing unit, etc., and the data processing unit can perform the steps described above with the stylus 101.

[0039] Terminal 102 is used to receive handwriting data and / or audio recording data sent by stylus 101, determine the text data corresponding to the audio recording data, and determine the third timestamp corresponding to each character in the text data based on each first timestamp. Based on the first timestamp, second timestamp, and third timestamp, it fuses multiple frames of audio data and multiple frames of handwriting images to obtain target fused data. Furthermore, terminal 102 can also display the target fused data on an interactive interface. Terminal 102 can be, but is not limited to, various personal computers, handwriting laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Terminal 102 can also install a client for the target fused data, which can display handwriting data, audio recording data, and target fused data.

[0040] In traditional technologies, the synchronization of audio recording and handwritten notes mainly includes the following categories: Traditional voice recorder and handwriting separation technology: Traditional voice recorders use digital recording technology to record audio, while users simultaneously write notes on paper. After recording, users need to manually associate the handwriting with the recording, which is tedious and prone to errors. Because the recording and handwriting exist completely independently with inconsistent time references, users find it difficult to accurately recall the handwriting location corresponding to each topic, resulting in extremely low recall efficiency; Dot matrix pen and recording linkage technology: Smart pen products such as Livescribe use dot matrix paper in conjunction with a pen tip camera to identify the dot matrix position, achieving the function of simultaneous handwriting capture and audio recording. The limitation of this type of product is that it must use special dot matrix paper, increasing the cost of use, and although the recording and handwriting data are time-aligned, they rely on special paper and cannot be adapted to ordinary paper, notebooks, and other everyday writing scenarios; Photo recording combined with recording technology: Some applications use mobile phones or cameras to photograph and archive paper handwriting, and then associate and store it with the audio file. However, this solution requires users to manually take photos, lacks an automatic correlation mechanism between handwriting and audio recording, has inconsistent time references, and requires manual matching during retrospection, resulting in low efficiency. Electromagnetic induction handwriting tablet recording solutions: Wacom and other companies' handwriting tablet products use electromagnetic induction technology to capture the position of the stylus, working with external recording devices to achieve simultaneous handwriting and audio recording. However, this type of solution requires a dedicated handwriting tablet, with the recording and handwriting devices completely separate, and independent time references, making it impossible to achieve accurate correlation between audio recording and handwriting.

[0041] Traditional technologies suffer from the following drawbacks: Firstly, the time bases for recording and handwriting are completely independent: Recording devices can only record audio, while handwriting recording devices can only capture handwriting. These two types of devices operate independently, with entirely different time bases. Even when recording and handwriting are started simultaneously, their start and end times differ, making precise matching impossible. Users often struggle to quickly locate corresponding recording segments using handwriting analysis during later review, requiring repeated listening and comparison, which is extremely inefficient. Secondly, there is a lack of a unified millisecond-level time synchronization mechanism: Recording and handwriting data are timestamped separately, with different timestamp sources (recording device clock vs. handwriting device clock), potentially resulting in errors of several seconds or even tens of seconds. This lack of precision means that recording and handwriting can only be roughly correlated, failing to achieve millisecond-level accuracy and hindering the precise review needs of scenarios such as meeting minutes and classroom handwriting. Thirdly, the mapping between strokes and audio segments is inaccurate: the association between recording and handwriting remains at the session or page level, unable to be refined to the mapping of individual strokes to corresponding audio segments. Users can only roughly know that a certain segment of handwriting corresponds to a certain section of the recording, but cannot precisely know what was being discussed at the time of writing each stroke; two-way positioning capability is lacking: recording and handwriting association technology usually only supports one-way operation, either finding the handwriting through the recording or finding the recording through the handwriting, and the positioning accuracy is low and the response speed is slow. Users cannot quickly locate the corresponding recording segment at any handwriting position, nor can they highlight the corresponding handwriting content in real time during recording playback; later review relies on manual comparison: due to the inconsistent and inaccurate time reference between recording and handwriting, users need to spend a lot of time recalling, comparing, and searching to find the recording segment corresponding to the content of interest during later review. The complete disconnect between recording and handwriting makes it difficult to locate important information, reducing the value of the recording.

[0042] To address this, this application automatically triggers handwriting collection after recording begins and automatically stops when recording ends, requiring no manual operation from the user throughout the process. The recording and handwriting are naturally and accurately linked, improving efficiency and user convenience. At the same time, by fusing the timestamps of the recording data, text data, and handwriting data, precise data fusion is achieved, improving the data fusion effect.

[0043] In one exemplary embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 The following steps are used as an example of the terminal in the example, including steps 201 to 203.

[0044] Step 201: Obtain audio recording data and handwriting data from the stylus.

[0045] During implementation, the terminal acquires audio data and handwriting data collected by the stylus; during execution, audio data can be collected by the terminal or by the stylus, while handwriting data can be collected by the stylus or by the data processing unit configured on the stylus; the stylus sends the collected handwriting data and / or audio data to the terminal; the terminal receives the audio data and the handwriting data from the stylus.

[0046] In this application, audio recording data refers to the sound data of a user's speech, or the sound data of a conversation between users. The audio recording data includes multiple frames of audio data, each including a first timestamp, which characterizes the time at which each frame of audio data begins and / or ends recording. Handwriting data refers to image data of a user's handwriting. This handwriting can be left on an electronic device using a capacitive pen, or on a writing medium such as paper using a stylus. The handwriting data includes multiple frames of handwriting images, each including a second timestamp, which characterizes the time at which each frame of handwriting image begins and / or ends writing. It should be noted that each frame of audio data can be the audio corresponding to each character, and this audio data can correspond to multiple first timestamps; each frame of handwriting image can be an image corresponding to each character, and this handwriting image can correspond to multiple second timestamps.

[0047] Step 202: Determine the text data corresponding to the recording data, and determine the third timestamp corresponding to each character in the text data based on each first timestamp.

[0048] During implementation, the terminal identifies the text data corresponding to the recording data, or converts the recording data into text data, and determines the third timestamp corresponding to each character in the text data based on the first timestamp corresponding to each handwriting image.

[0049] During execution, the terminal can determine the text data corresponding to the recorded data through the accessed model. The terminal can write the recorded data into the model for text recognition and obtain the text data output by the model. The terminal can also determine the text data corresponding to the recorded data through a speech recognition algorithm. The terminal can run the speech recognition algorithm to obtain the text data corresponding to the recorded data.

[0050] In determining the third timestamp, since the writing speed is usually slower than the recording speed, that is, the number of words written in the same amount of time is less than the number of words spoken, the text data corresponding to the recording data can be matched with the handwriting data. To this end, the third timestamp corresponding to each character in the text data can be determined first based on the first timestamp corresponding to the speech data. The third timestamp is used to characterize the start time and / or end time corresponding to the recording of each character.

[0051] Step 203: Based on the first timestamp, the second timestamp, and the third timestamp, the multi-frame speech data and the multi-frame handwriting image are fused to obtain the target fused data.

[0052] During implementation, the terminal uses the third timestamp of each character in the text data as intermediate data. It performs time-series calibration by aligning the third timestamp with the first and second timestamps, and then fuses the timestamps corresponding to each character based on the aligned timestamps to obtain the target fused data.

[0053] During execution, the terminal can first construct multiple data pairs containing corresponding voice data and handwriting images. These data pairs are then fused to obtain the target fused data. It should be noted that since writing speed is usually slower than speaking speed, one frame of handwriting image may correspond to multiple frames of voice data; a data pair may include one frame of handwriting image and multiple frames of voice data. It is important to understand that the matching process for the first, second, and third timestamps is not merely selecting and combining timestamps with the same moment; rather, it involves matching corresponding voice data and handwriting data. For example, if n characters are written while speaking this passage, the first timestamp corresponding to this passage can be matched with the second timestamps corresponding to those n characters. Therefore, these n frames of handwriting images can be matched with the multiple frames of voice data contained in this passage.

[0054] In this application, the target fusion data can be multi-dimensional interactive data that includes audio recording data, handwriting data, and / or text data corresponding to the audio recording data. Specifically, it can be a handwriting page / interactive page that includes multiple interactive controls. The interactive controls can be configured on each text in the page. The terminal can display the target fusion data, and the user can interact with the displayed target fusion data.

[0055] During the display of target fused data, the terminal can display the timeline corresponding to the handwriting data and / or audio data included in the target fused data. The user can trigger the handwriting data corresponding to any text on the target fused data display page, and the terminal can respond to the trigger command to play the audio data corresponding to the handwriting data. Alternatively, the user can drag the timeline corresponding to the audio data to select the display time, and the terminal can respond to the trigger command to highlight the handwriting data corresponding to the display time.

[0056] In the above data processing method, audio data and / or handwriting data are collected by a stylus and sent to the terminal. The audio data includes multiple frames of voice data, each of which includes a first timestamp. The handwriting data includes multiple frames of handwriting images, each of which includes a second timestamp. The terminal performs text-to-text processing on the audio data to obtain the corresponding text data. Based on the first timestamp, a third timestamp is determined for each character in the text data. The multiple frames of voice data and multiple frames of handwriting images are fused based on the first, second, and third timestamps to obtain the target fused data. Through the cooperation of the stylus and the terminal, the timestamps of the audio data and handwriting data are aligned, thereby ensuring the temporal consistency of the audio data and handwriting data. This allows the handwriting and audio data to be synchronized when the target fused data is displayed, improving the fusion effect of the audio data and handwriting data.

[0057] Based on the above exemplary embodiment, the following provides a data processing method in one or more exemplary embodiments, which can be applied to a terminal. Figure 1 Taking the terminal in the example, the explanation includes the following content.

[0058] In the process of acquiring audio and handwriting data, audio and handwriting data can be collected using a stylus, and timestamps can be added to both using the stylus; in one optional implementation provided by this application, such as Figure 3 As shown, step 201 includes step 301:

[0059] Step 301: Receive the audio data and handwriting data sent by the stylus.

[0060] During implementation, the user inputs a recording command into the stylus, either through the stylus's button or the terminal's interactive page. The stylus collects recording data through the voice acquisition component and handwriting data through the image acquisition component. While collecting data, it timestamps each frame of handwriting image and each frame of voice data based on the first and second timestamps corresponding to the current moment, until a recording end command is received, thus obtaining the recording data and handwriting data. After obtaining the recording data and handwriting data, the user sends the recording data and handwriting data to the terminal; the terminal receives the recording data and handwriting data sent by the stylus, which are accompanied by timestamps.

[0061] During timestamp marking, the stylus uses the high-precision real-time clock (RTC) built into the main control chip as the sole clock source to provide a unified time reference. Furthermore, a temperature compensation circuit can be incorporated to minimize clock drift after prolonged use. During execution, the stylus assigns a second timestamp to each captured handwriting image frame and a first timestamp to each segment of audio data via the main control chip. The timestamp format is a 64-bit unsigned integer in milliseconds, with the timestamp zero occurring at the moment the user presses the record button. All subsequent timestamps increment based on the same clock source, ensuring perfect alignment between the handwriting timeline and the recording timeline.

[0062] During the process of collecting handwriting data, the stylus can capture images at a preset frame rate and can also compress the captured images to save storage space.

[0063] The audio data and handwriting data are obtained by the stylus responding to the recording command and, based on the first and second timestamps corresponding to the current moment, by timestamping each frame of handwriting image and each frame of audio data.

[0064] One optional implementation provided in this application uses a stylus to timestamp handwriting images and voice data based on a unified clock source, ensuring the synchronization of the timestamps of the two at every moment, thereby ensuring the temporal reliability of data fusion.

[0065] In the process of acquiring audio and handwriting data, to alleviate the data processing pressure on a single device, handwriting and audio data can be acquired separately using two devices; another optional implementation provided in this application is as follows: Figure 4 As shown, step 201 includes steps 401 to 403:

[0066] Step 401: In response to the recording command, the audio acquisition component acquires recording data and sends a handwriting acquisition command to the stylus.

[0067] In real-world scenarios, a stylus can be used to collect handwriting data / images while the terminal collects audio data. During use, the user can input recording commands on the terminal's recording page. After receiving the recording command, the terminal needs to synchronize the handwriting collection command to the stylus to instruct it to collect handwriting data. However, time synchronization issues can easily arise when two devices collect data. Therefore, while collecting handwriting data, the stylus can also send a time calibration request to the terminal. The terminal can send the first timestamp of the audio data to the stylus so that it can automatically calibrate the second timestamp. Alternatively, the terminal can directly send the second timestamp to the stylus so that it can directly use the second timestamp to ensure that the timestamps of the audio data and handwriting data are synchronized.

[0068] During implementation, the terminal responds to the input recording command by calling the audio acquisition component to collect recording data. At the same time, it sends a handwriting acquisition command to the stylus so that the stylus can simultaneously perform handwriting acquisition.

[0069] Step 402: In response to the time calibration request sent by the stylus, send the first timestamp and / or the second timestamp corresponding to the current time to the stylus.

[0070] During implementation, the stylus can periodically send time calibration requests to the terminal during the handwriting image acquisition process for timestamp calibration. It can send the first timestamp of the voice data marked at the current moment to the stylus, or send the second timestamp corresponding to the handwriting data at the current moment to the handwriting tablet.

[0071] During execution, the stylus sends a calibration request to the terminal. The data packet contains the stylus's current timestamp T_A. Upon receiving the request, the terminal records its local timestamp T_B_recv at the moment of receipt and encapsulates its own current timestamp T_B in the response data packet. Upon receiving the response, the stylus records its local timestamp T_A_recv at the moment of receipt. The stylus calculates the network latency estimate and clock offset, and corrects all subsequent timestamps based on the calculated offset value. Furthermore, a heartbeat packet is sent to the terminal, containing the stylus's current timestamp. Upon receiving the heartbeat packet, the terminal converts the stylus's timestamp into its equivalent timestamp based on the clock offset correction value. The terminal compares the corrected timestamp with its local timestamp, calculates the new error, and updates the offset value. Through real-time correction, the timeline error between the two devices is always kept within 1 millisecond during recording.

[0072] It should be noted that step 402 will be executed repeatedly during the recording process until the recording ends, that is, until the recording ends command is received.

[0073] Step 403: In response to the recording end command, receive the handwriting data sent by the stylus, and save the handwriting data and recording data if the first timestamp and the second timestamp match.

[0074] During implementation, the terminal receives and responds to the externally input recording end command and synchronizes the recording end command to the stylus; the stylus sends the integrated handwriting data to the terminal; the terminal receives the handwriting data sent by the stylus and performs time calibration matching between the first timestamp and the second timestamp. If the matching is successful, the handwriting data and the recording data are saved.

[0075] One optional implementation provided in this application improves the reliability of timestamps by synchronizing and monitoring timestamps between two devices, and further enhances the reliability of data fusion by providing timestamps based on only one device.

[0076] It should be noted that, in the process of timestamp marking, this application employs the following safeguards to ensure time synchronization accuracy: NTP clock calibration: The device supports clock calibration via Network Time Protocol (NTP), and can periodically synchronize with an authoritative time server in a network environment, ensuring that clock accuracy does not drift after long-term use; Temperature compensation: The clock module integrates a temperature compensation circuit to suppress the impact of temperature changes on clock accuracy, maintaining clock accuracy within ±10ppm in a temperature range of -20°C to +50°C; Power failure protection: The clock module is independently powered, and the clock can continue to run after the device is powered off, ensuring that the timeline is continuous after restarting without time rollback or jumps; Delay compensation: The main control chip records the processing delay of each module and performs compensation and correction during data association, eliminating synchronization errors caused by end-to-end delays from acquisition to storage; Dual-device heartbeat mechanism: A stable heartbeat connection is maintained between the two devices, and recalibration is automatically triggered when a clock offset exceeds a threshold.

[0077] In the process of data fusion, multiple data groups can be constructed by first selecting matching handwriting images and voice data based on timestamps, and then fusion of multiple data groups to obtain the target fused data; in one optional implementation method provided in this application, such as Figure 5 As shown, step 203 includes steps 501 to 502:

[0078] Step 501: For each second timestamp, traverse the target third timestamp that matches the second timestamp in each third timestamp, determine the target first timestamp that matches the target third timestamp in each first timestamp, and combine the voice data corresponding to the target first timestamp and the handwriting image corresponding to the second timestamp to obtain a data group.

[0079] During implementation, the terminal can perform mutual mapping between voice data and handwriting images based on text data. It can also perform matching based on the third timestamp of the text data. For each second timestamp (i.e., for the second timestamp corresponding to each handwriting image), the terminal iterates through the target third timestamps that match the second timestamps in each third timestamp set. This means iterates through the timestamps in the text data that match the handwriting images. Finally, it determines the target first timestamp that matches the target third timestamp in each first timestamp set. This means it determines the timestamp in the recording data that matches the text data. The terminal then combines the voice data corresponding to the target first timestamp with the handwriting image corresponding to the second timestamp to obtain a data set. This is because the recording data is stored in digital / analog audio format, making direct matching between audio and images impossible. Therefore, it requires the use of text data as intermediate data for matching.

[0080] During execution, the terminal first obtains the start and end timestamps [stroke_start, stroke_end] of each frame of handwriting image, and then obtains the audio data of each frame. It then traverses all recording segments, finds segments that intersect with the start and end timestamp range, calculates the proportion of time covering the segment for each intersecting segment, selects the recording segment with the highest coverage proportion as the main associated segment of the handwriting image, associates the recording segment with the handwriting image, and constructs a data group.

[0081] Furthermore, multiple frames of speech data and multiple frames of handwriting images can be fused based on the first, second, and third timestamps and text data to obtain target fused data; the characters corresponding to the handwriting images can be identified, and the identified characters can be matched with the characters contained in the text data. When both the timestamp and the characters match, the speech data corresponding to the first timestamp and the handwriting images corresponding to the second timestamp are combined to obtain a data set.

[0082] Step 502: Merge the data groups to obtain the target fused data.

[0083] During implementation, the terminal fuses data based on each data group to obtain the target fused data. In this application, the target fused data can be triggered on the interactive page, after which the corresponding audio recording data can be played, or handwriting data can be displayed.

[0084] One optional implementation provided in this application uses text data as intermediate data to connect audio recording data and handwriting data. By matching timestamps, it accurately and reliably associates each frame of audio data and each frame of handwriting image, thereby improving the reliability of data fusion and thus improving the usability of the target fused data.

[0085] During data fusion, to improve the convenience of using the target fused data, index tags can be generated for each data group to facilitate the location of each data group; one optional implementation method provided in this application is as follows: Figure 6 As shown, step 502 includes steps 601 to 602:

[0086] Step 601: Determine the index marker for each data group based on the temporal order of each second timestamp in the handwriting data.

[0087] During implementation, the terminal establishes index tags based on handwriting images as units and the temporal sequence of the handwriting images as the standard. The terminal can determine the index tags of each data group according to the temporal sequence of the second timestamp of each frame of handwriting images in the handwriting data. This is because each data group contains handwriting images, and each frame of handwriting image may correspond to multiple frames of voice data; therefore, it is chosen to construct index tags based on handwriting images as units.

[0088] Step 602: Merge each data group according to each index mark to obtain the target fused data.

[0089] During implementation, the terminal marks each data group according to each index tag, and then merges the marked data groups according to the tag order to obtain the target fused data.

[0090] During execution, the terminal can also create a page ID → page start and end time → all handwriting images and audio data contained therein, on a page-by-page basis; or it can create a chapter ID → chapter start and end time → all pages contained therein, on a chapter-by-chapter basis.

[0091] One optional implementation provided in this application involves marking data groups with index data and then fusing the marked data groups to obtain target fused data. This allows the terminal to quickly query data using the index markings when using the target fused data. At the same time, the index markings can also accurately sort the data groups, thereby obtaining reliable target fused data.

[0092] During the use of target fusion data, the user can trigger handwriting within the target fusion data to play audio data from the timestamp corresponding to the handwriting image; in one optional implementation provided by this application, such as Figure 7 As shown, the method further includes step 701:

[0093] Step 701: In response to a trigger command for the first handwriting image in the target fused data, query the target index marker corresponding to the first handwriting image, determine the first timestamp corresponding to the target index marker, and jump to play the first audio data at the first timestamp corresponding to the target index marker. Here, the first handwriting image can be a handwriting image corresponding to a certain text in the target fused data. Correspondingly, the first audio data can be the audio data corresponding to the first handwriting image, or the speech data recorded by the user when recording the first handwriting image. The user triggering the first handwriting image means wanting the terminal to display the audio data corresponding to the first handwriting image, or the audio data recorded when recording the first handwriting image.

[0094] During implementation, the terminal displays the target fusion data on the interactive page; the user submits a trigger command on the interactive page for the first handwriting image in the target fusion data; in response to the trigger command, the terminal queries the target index mark corresponding to the first handwriting image and determines the first timestamp corresponding to the target index mark, that is, the first timestamp at the start of writing of the first handwriting image, jumps to the first timestamp to play the first voice data, and continues to play subsequent multiple frames of voice data from the first voice data.

[0095] In another alternative implementation, the terminal can display the corresponding handwriting image by dragging the progress bar of the recorded data, such as... Figure 8 As shown, the method also includes step 801:

[0096] Step 801: In response to a trigger command for the first voice data in the target fused data, query the target index marker corresponding to the first voice data, determine the second timestamp corresponding to the target index marker, and display the first handwriting image of the second timestamp corresponding to the target index marker. The first voice data can be recording data corresponding to a specific moment in the target fused data. Correspondingly, the first handwriting image can be a handwriting image corresponding to the first voice data, or a handwriting image recorded at the moment the user triggers the command. The user triggering the first voice data means they want the terminal to display the handwriting image corresponding to the voice data at that moment, or the handwriting image of the text recorded at that moment.

[0097] During implementation, the terminal displays the target fusion data on the interactive page; the user submits a trigger command for the first voice data in the target fusion data on the interactive page; the terminal responds to the trigger command, queries the target index mark corresponding to the first voice data, and determines the second timestamp corresponding to the target index mark, that is, the second timestamp at the start time of recording the first voice data, jumps to the first handwriting image corresponding to the second timestamp and displays it.

[0098] The two optional implementation methods provided in this application allow users to directly trigger the target fusion data to jump to and display audio / handwriting data. The terminal can determine the content to be displayed based on the index markers according to the trigger command submitted by the user, which greatly improves the convenience of the user. At the same time, the accuracy of the displayed content is ensured based on the index markers.

[0099] In one embodiment, see Figure 9 It illustrates an interaction timing diagram of a data processing method provided in an embodiment of this application, which can be applied to... Figure 1 In the terminal shown. For example... Figure 9 As shown, the data processing method may include the following steps:

[0100] Step 905: Receive audio data and handwriting data sent by the stylus.

[0101] Step 906: Determine the text data corresponding to the recording data, and determine the third timestamp corresponding to each character in the text data based on each first timestamp.

[0102] Step 907: For each second timestamp, iterate through the target third timestamps that match the second timestamps in each third timestamp, and determine the target first timestamps that match the target third timestamps in each first timestamp.

[0103] Step 908: Combine the voice data corresponding to the first timestamp of the target and the handwriting image corresponding to the second timestamp to obtain a data set.

[0104] Step 909: Determine the index marker for each data group based on the temporal order of each second timestamp in the handwriting data.

[0105] Step 910: Merge each data group according to each index mark to obtain the target fused data.

[0106] Step 911: In response to the trigger command for the first handwriting image in the target fused data, query the target index mark corresponding to the first handwriting image, determine the first timestamp corresponding to the target index mark, and jump to play the first voice data corresponding to the first timestamp of the target index mark.

[0107] It should be noted that any one or more of steps 905 to 911, or any combination of steps 201 to 203 provided in the above embodiments, can be selected to form a new implementation method according to the needs of implementation and deployment. Furthermore, any one or more technical features in the technical solution composed of steps 905 to 911 can also be selected to form a new implementation method according to the actual deployment needs, or technical features in one or more optional implementations provided in one or more of the above embodiments can be selected to form a new implementation method. These will not be elaborated on here.

[0108] It should be noted that the steps in one or more of the above embodiments can be performed by a terminal, and the steps in one or more of the following embodiments can be performed by a stylus. The two can be referenced each other. The steps and limitations in one or more of the above embodiments are also described in one or more of the following embodiments. Correspondingly, the steps and limitations in one or more of the following embodiments are also described in one or more of the above embodiments. Therefore, they will not be repeated here.

[0109] In one exemplary embodiment, such as Figure 10 As shown, another data processing method is provided, which can be applied to... Figure 1 The following steps are used as an example to illustrate the process of using a stylus pen.

[0110] Step 1001: Obtain the target data.

[0111] During implementation, the stylus responds to externally inputted data acquisition commands and acquires the target data. During execution, the user can trigger the data acquisition command by pressing a button configured on the stylus, or the user can send the data acquisition command to the stylus via a client installed on the terminal. The stylus can acquire the target data after detecting a recording command / handwriting capture command.

[0112] In this application, the target data includes audio recording data and / or handwriting data. The audio recording data includes multiple frames of voice data, each of which includes a first timestamp. The handwriting data includes multiple frames of handwriting images, each of which includes a second timestamp.

[0113] Step 1002: Send the target data to the terminal.

[0114] During implementation, the stylus sends the target data to the terminal after acquiring it. During execution, the stylus can also send the target data to the terminal after detecting a recording end command. Accordingly, the target data is used by the terminal to fuse multiple frames of audio data and multiple frames of handwriting images based on a first timestamp, a second timestamp, and a third timestamp. The third timestamp is the timestamp corresponding to each character in the text data corresponding to the audio data. The specific execution steps on the terminal side can be referred to one or more of the above embodiments, and will not be repeated here.

[0115] In the above data processing method, audio data and / or handwriting data are collected by a stylus and sent to the terminal. The audio data includes multiple frames of voice data, each of which includes a first timestamp. The handwriting data includes multiple frames of handwriting images, each of which includes a second timestamp. The terminal performs text-to-text processing on the audio data to obtain the corresponding text data. Based on the first timestamp, a third timestamp is determined for each character in the text data. The multiple frames of voice data and multiple frames of handwriting images are fused based on the first, second, and third timestamps to obtain the target fused data. Through the cooperation of the stylus and the terminal, the timestamps of the audio data and handwriting data are aligned, thereby ensuring the temporal consistency of the audio data and handwriting data. This allows the handwriting and audio data to be synchronized when the target fused data is displayed, improving the fusion effect of the audio data and handwriting data.

[0116] In real-world scenarios, a stylus can simultaneously acquire audio recording data and handwriting data; in one optional implementation provided in this application, such as Figure 11 As shown, step 1001 includes steps 1101 to 1103:

[0117] Step 1101: In response to the recording command, acquire multiple frames of candidate handwriting images through the image acquisition component and acquire multiple frames of candidate speech data through the audio acquisition component.

[0118] During implementation, the stylus responds to the recording command by acquiring multiple frames of candidate handwriting images through the image acquisition component and acquiring multiple frames of candidate speech data through the audio acquisition component. The specific acquisition process can be referred to one or more of the above embodiments, and will not be repeated here.

[0119] Step 1102: For each frame of candidate handwriting image and each frame of candidate speech data, the candidate speech data is timestamped based on the first timestamp corresponding to the current time to obtain speech data, and the candidate handwriting image is timestamped based on the second timestamp corresponding to the current time to obtain handwriting image.

[0120] In the implementation process, the stylus timestamps the candidate speech data for each frame of candidate handwriting image and each frame of candidate speech data based on the first timestamp corresponding to the current time to obtain speech data, and timestamps the candidate handwriting image based on the second timestamp corresponding to the current time to obtain handwriting image. The specific marking process can be referred to one or more of the above embodiments, and will not be repeated here.

[0121] Step 1103: In response to the recording end command, combine the voice data in sequence to obtain the recording data, and combine the handwriting images in sequence to obtain the handwriting data.

[0122] During implementation, the stylus responds to the recording end command, combines the voice data in sequence to obtain the recording data, and combines the handwriting images in sequence to obtain the handwriting data.

[0123] One optional implementation provided in this application involves collecting audio recording data and handwriting data using a stylus and then timestamping them. The reliability and stability of the timestamps are ensured by using a single clock on a single device, thereby guaranteeing the availability of the data fusion.

[0124] In one embodiment, see Figure 9 It illustrates an interaction timing diagram of a data processing method provided in an embodiment of this application, which can be applied to... Figure 1 The stylus shown. For example... Figure 9 As shown, the data processing method may include the following steps:

[0125] Step 901: In response to the recording command, acquire multiple frames of candidate handwriting images through the image acquisition component and acquire multiple frames of candidate speech data through the audio acquisition component.

[0126] Step 902: For each candidate handwriting image and each candidate speech data, the candidate speech data is timestamped based on the first timestamp corresponding to the current time to obtain speech data, and the candidate handwriting image is timestamped based on the second timestamp corresponding to the current time to obtain handwriting image.

[0127] Step 903: In response to the recording end command, combine the voice data in sequence to obtain the recording data, and combine the handwriting images in sequence to obtain the handwriting data.

[0128] Step 904: Send the audio recording data and handwriting data to the terminal.

[0129] It should be added that, when recording, the terminal and / or voice recorder provided in this application can initialize a session ID (UUID), then set the session start timestamp = T=0 (the time the record button is pressed), create an empty handwriting data queue and a recording data queue, establish a session index structure (empty), start the handwriting acquisition thread (starting from T=0), start the recording acquisition thread (starting from T=0), and register the session to the active session list for convenient management of recording sessions. When recording ends, the handwriting acquisition thread can be stopped, followed by the recording acquisition thread, and all remaining handwriting queue data and recording queue data can be written to storage. A complete stroke-to-sentence mapping relationship can be established, session metadata (total duration, number of strokes, number of recording segments, etc.) can be generated, the session status can be marked as "completed," and a background task (optional) can be started: uploading the data to the cloud for OCR / ASR processing.

[0130] In addition, this application can also pause the recording process. When the terminal and / or the recorder receives a pause command, the last synchronization time point T_checkpoint can be recorded. When the recording resumes, new data continues to be collected from T_checkpoint. New handwriting frames and new audio frames are associated within the time range after T_checkpoint. Already associated data is skipped to avoid repeated association operations. After the association is completed, T_checkpoint is updated.

[0131] It should be noted that, although the technical solutions of this application are mainly described using direct interaction between local terminals (such as a pen and a terminal) as an example in one or more embodiments, this does not constitute a limitation on the implementation scenarios of this application. Those skilled in the art will understand that some or all of the data processing, instruction generation, information recognition, and synchronization operations described in this application are not limited to execution by local devices; they can also be executed collaboratively or independently by cloud servers, backend servers, edge computing nodes, software applications (Apps), web clients, or third-party service platforms that are communicatively connected to the local terminal. That is, the core interaction logic of this application can be flexibly deployed based on a client-server architecture or a cloud service architecture, and the specific task allocation among the various execution entities can be adaptively adjusted according to the actual network environment, computing resources, and business needs; all of these fall within the scope of protection covered by this application.

[0132] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0133] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0134] In one exemplary embodiment, such as Figure 12 As shown, a data processing device is provided, including: a data receiving module 1201, a timing module 1202, and a data fusion module 1203, wherein: the data receiving module 1201 is used to acquire audio recording data and handwriting data from a stylus; the audio recording data includes multiple frames of audio data, each audio data including a first timestamp; the handwriting data includes multiple frames of handwriting images, each handwriting image including a second timestamp; the timing module 1202 is used to determine the text data corresponding to the audio recording data, and determine the third timestamp corresponding to each character in the text data based on each first timestamp; the data fusion module 1203 is used to fuse the multiple frames of audio data and the multiple frames of handwriting images based on the first timestamp, the second timestamp, and the third timestamp to obtain target fused data.

[0135] In one embodiment, the data receiving module 1201 includes a data receiving unit for receiving audio recording data and handwriting data sent by a stylus.

[0136] In one embodiment, the data receiving module 1201 includes a recording unit, a time calibration unit, and a data storage unit, wherein: the recording unit is used to collect recording data through an audio acquisition component in response to a recording command, and send a handwriting acquisition command to the stylus; the time calibration unit is used to send a first timestamp and / or a second timestamp corresponding to the current time to the stylus in response to a time calibration request sent by the stylus; the data storage unit is used to receive handwriting data sent by the stylus in response to a recording end command, and save the handwriting data and the recording data if each first timestamp matches each second timestamp.

[0137] In one embodiment, the data fusion module 1203 includes a traversal unit and a fusion unit, wherein: the traversal unit is used to traverse the target third timestamp that matches the second timestamp in each third timestamp for each second timestamp, determine the target first timestamp that matches the target third timestamp in each first timestamp, and combine the voice data corresponding to the target first timestamp and the handwriting image corresponding to the second timestamp to obtain a data group; the fusion unit is used to fuse the data groups to obtain the target fused data.

[0138] In one embodiment, the fusion unit includes an index determination unit and an index marking unit, wherein: the index determination unit is used to determine the index marking of each data group based on the temporal order of each second timestamp in the handwriting data; the index marking unit is used to fuse each data group according to each index marking to obtain target fused data.

[0139] In one embodiment, the device further includes a first display unit, configured to, in response to a trigger command for a first handwriting image in the target fused data, query a target index marker corresponding to the first handwriting image, determine a first timestamp corresponding to the target index marker, and jump to play the first voice data corresponding to the first timestamp of the target index marker.

[0140] In one embodiment, the apparatus further includes a second display unit, configured to, in response to a trigger command for the first voice data in the target fused data, query the target index marker corresponding to the first voice data, determine the second timestamp corresponding to the target index marker, and display the first handwriting image of the second timestamp corresponding to the target index marker.

[0141] Based on the same inventive concept, this application also provides another data processing apparatus for implementing the other data processing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method above, and will not be repeated here.

[0142] In one exemplary embodiment, such as Figure 13 As shown, a data processing device is provided, including: a data acquisition module 1301 and a data transmission module 1302, wherein: the data acquisition module 1301 is used to acquire target data, the target data including audio recording data and / or handwriting data, the audio recording data including multiple frames of audio data, each audio data including a first timestamp, the handwriting data including multiple frames of handwriting images, each handwriting image including a second timestamp; the data transmission module 1302 is used to send the target data to a terminal, the target data being used by the terminal to fuse the multiple frames of audio data and the multiple frames of handwriting images based on the first timestamp, the second timestamp, and a third timestamp to obtain target fused data, the third timestamp being the timestamp corresponding to each character in the text data corresponding to the audio recording data.

[0143] In one embodiment, the data acquisition module 1301 includes a data acquisition unit, a data marking unit, and a data combination unit, wherein: the data acquisition unit is used to acquire multiple frames of candidate handwriting images through an image acquisition component and acquire multiple frames of candidate speech data through an audio acquisition component in response to a recording command; the data marking unit is used to timestamp the candidate speech data based on a first timestamp corresponding to the current time for each frame of candidate handwriting image and each frame of candidate speech data to obtain speech data, and to timestamp the candidate handwriting image based on a second timestamp corresponding to the current time to obtain handwriting images; the data combination unit is used to combine each speech data in time sequence to obtain recording data in response to a recording end command, and to combine each handwriting image in time sequence to obtain handwriting data.

[0144] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0145] In one exemplary embodiment, a computer device is provided, which may be a terminal or a stylus, and its internal structure diagram may be as follows. Figure 14As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a data processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0146] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0147] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the embodiments corresponding to one data processing method and / or the steps of the embodiments corresponding to another data processing method.

[0148] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the embodiments corresponding to one data processing method and / or the steps of the embodiments corresponding to another data processing method.

[0149] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the embodiments corresponding to one data processing method and / or the steps of the embodiments corresponding to another data processing method.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0151] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0152] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0153] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, include: Acquire audio recording data and handwriting data from a stylus. The audio recording data includes multiple frames of voice data, each of which includes a first timestamp. The handwriting data includes multiple frames of handwriting images, each of which includes a second timestamp. Determine the text data corresponding to the recording data, and determine the third timestamp corresponding to each character in the text data based on each of the first timestamps; The multi-frame audio data and the multi-frame handwriting images are fused based on the first timestamp, the second timestamp, and the third timestamp to obtain the target fused data.

2. The method according to claim 1, characterized in that, The acquisition of audio recording data and handwriting data includes: Receive the audio recording data and the handwriting data sent by the stylus; The audio recording data and the handwriting data are obtained by the stylus responding to the recording command by timestamping each frame of the handwriting image and each frame of the audio data based on the first timestamp and the second timestamp corresponding to the current time.

3. The method according to claim 1, characterized in that, The acquisition of audio recording data and handwriting data includes: In response to a recording command, the audio acquisition component acquires the recording data, and sends a handwriting acquisition command to the stylus. In response to a time calibration request sent by the stylus, a first timestamp and / or a second timestamp corresponding to the current moment are sent to the stylus; and In response to a recording end command, the handwriting data sent by the stylus is received, and the handwriting data and the recording data are saved if each of the first timestamps and each of the second timestamps match.

4. The method according to any one of claims 1 to 3, characterized in that, The process of fusing the multi-frame speech data and the multi-frame handwriting images based on the first timestamp, the second timestamp, and the third timestamp to obtain target fused data includes: For each second timestamp, traverse the target third timestamps that match the second timestamps in each of the third timestamps, determine the target first timestamps that match the target third timestamps in each of the first timestamps, and combine the voice data corresponding to the target first timestamps and the handwriting image corresponding to the second timestamps to obtain a data group; and The target fused data is obtained by fusing the data groups.

5. The method according to claim 4, characterized in that, The fusion of each of the data groups to obtain the target fused data includes: Based on the temporal order of each of the second timestamps in the handwriting data, determine the index marker for each of the data groups; and The data groups are fused according to the index tags to obtain the target fused data.

6. The method according to claim 5, characterized in that, Also includes: In response to a trigger command for a first handwriting image in the target fused data, the system queries the target index marker corresponding to the first handwriting image, determines the first timestamp corresponding to the target index marker, and jumps to play the first voice data corresponding to the first timestamp of the target index marker. or In response to a trigger command for the first voice data in the target fused data, the system queries the target index marker corresponding to the first voice data, determines the second timestamp corresponding to the target index marker, and displays the first handwriting image of the second timestamp corresponding to the target index marker.

7. A data processing method, characterized in that, include: Acquire target data, the target data including audio recording data and / or handwriting data, the audio recording data including multiple frames of voice data, each of the voice data including a first timestamp, the handwriting data including multiple frames of handwriting images, each of the handwriting images including a second timestamp; The target data is sent to the terminal. The target data is used by the terminal to fuse the multi-frame voice data and the multi-frame handwriting image based on the first timestamp, the second timestamp, and the third timestamp to obtain target fused data. The third timestamp is the timestamp corresponding to each character in the text data corresponding to the recording data.

8. The method according to claim 7, characterized in that, The acquisition of target data includes: In response to a recording command, multiple frames of candidate handwriting images are acquired through the image acquisition component, and multiple frames of candidate speech data are acquired through the audio acquisition component. For each frame of candidate handwriting image and each frame of candidate speech data, the candidate speech data is timestamped based on the first timestamp corresponding to the current time to obtain the speech data, and the candidate handwriting image is timestamped based on the second timestamp corresponding to the current time to obtain the handwriting image; and In response to a recording end command, the audio data is obtained by combining the audio data in a time sequence, and the handwriting data is obtained by combining the handwriting images in a time sequence.

9. A data processing apparatus, characterized in that, The device includes: The data receiving module is used to acquire audio data and handwriting data from a stylus. The audio data includes multiple frames of voice data, each of which includes a first timestamp. The handwriting data includes multiple frames of handwriting images, each of which includes a second timestamp. The timing module is used to determine the text data corresponding to the recording data, and to determine the third timestamp corresponding to each character in the text data based on each of the first timestamps; The data fusion module is used to fuse the multi-frame speech data and the multi-frame handwriting images based on the first timestamp, the second timestamp, and the third timestamp to obtain target fused data.

10. A data processing apparatus, characterized in that, The device includes: A data acquisition module is used to acquire target data, the target data including audio recording data and / or handwriting data, the audio recording data including multiple frames of audio data, each of the audio data including a first timestamp, and the handwriting data including multiple frames of handwriting images, each of the handwriting images including a second timestamp. The data sending module is used to send the target data to the terminal. The target data is used by the terminal to fuse the multi-frame voice data and the multi-frame handwriting image based on the first timestamp, the second timestamp, and the third timestamp to obtain target fused data. The third timestamp is the timestamp corresponding to each character in the text data corresponding to the recording data.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6 or 7 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6 or 7 to 8.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6 or 7 to 8.

Images