An audio and written content joint processing method, a smart pen and a smart terminal

Abstract

The application relates to an audio and written content combined processing method, an intelligent pen and an intelligent terminal. The intelligent pen comprises a camera and an audio acquisition component, and is internally provided with a hardware clock source. The intelligent pen is in communication connection with the intelligent terminal, and the method comprises the following steps: acquiring written content image data collected by the camera, and marking the written content image data with a data timestamp based on the hardware clock source; acquiring audio data collected by the audio acquisition component, and marking the audio data with an audio timestamp based on the hardware clock source; and sending the written content image data, the data timestamp, the audio data and the audio timestamp to the intelligent terminal, instructing the intelligent terminal to recognize text of the written content image data, associating the recognized text content with the audio data based on respective timestamps, and generating a time synchronization association relationship between the text content and the audio data. Through the application, handwritten content can be recorded on any writing medium, and the association between recording and handwriting recording can be completed.

Description

Technical Field

[0001] This application relates to the field of recording technology, and in particular to a method for joint processing of audio and written content, a smart pen, and a smart terminal. Background Technology

[0002] With the increasing popularity of multi-person interactive scenarios such as meetings, classrooms, and interviews, people's dual demand for handwritten notes and audio recording is growing.

[0003] In related technologies, key information is typically recorded by handwriting while a smart pen is used to record the entire audio. However, the handwritten notes and the audio recordings are independent of each other, requiring repeated comparisons when organizing the notes, which is inefficient.

[0004] Therefore, how to link audio recordings and handwritten notes has become an urgent problem to be solved. Summary of the Invention

[0005] Therefore, it is necessary to provide a method for joint processing of audio and written content, a smart pen, and a smart terminal to address the aforementioned technical problems.

[0006] Firstly, this application provides a method for jointly processing audio and written content, applied to a smart pen. The smart pen includes a camera and an audio acquisition component, and has a built-in hardware clock source. The smart pen is communicatively connected to a smart terminal, including:

[0007] Acquire image data of the written content captured by the camera, and timestamp the image data of the written content based on the hardware clock source;

[0008] Acquire audio data collected by the audio acquisition component and timestamp the audio data based on a hardware clock source; and

[0009] The system sends the written content image data and corresponding data timestamps, as well as the audio data and corresponding audio timestamps, to the smart terminal. The smart terminal is instructed to perform text recognition on the written content image data and associate the recognized text content with the audio data based on their respective timestamps to generate a time synchronization relationship between the text content and the audio data.

[0010] In one embodiment, the image data of the written content is timestamped based on a hardware clock source, including:

[0011] Frame-by-frame recognition of the collected multi-frame image data of written content; and

[0012] If the similarity between the written content image data of the current frame and the written content image data of the previous frame is less than a preset similarity threshold, the current frame is determined as the target frame, and the written content image data corresponding to the target frame is timestamped according to the hardware clock source.

[0013] In one embodiment, audio data is timestamped based on a hardware clock source, including:

[0014] Multiple audio data segments are acquired based on a preset audio acquisition step size, and each audio data segment is timestamped based on a hardware clock source.

[0015] Secondly, this application also provides a method for joint processing of audio and written content, applied to a smart terminal, wherein the smart terminal is communicatively connected to the smart pen described above, and the method includes:

[0016] The system acquires image data of the written content, data timestamp, audio data, and audio timestamp sent by the smart pen. The image data of the written content is captured by the smart pen's camera. The data timestamp is obtained by timestamping the image data of the written content based on the hardware clock source built into the smart pen. The audio data is captured by the audio acquisition component of the smart pen. The audio timestamp is obtained by timestamping the audio data based on the hardware clock source.

[0017] Based on the written content image data, data timestamp, audio data, and audio timestamp, generate the joint processing result of the written content image data and audio data.

[0018] In one embodiment, the written content image data, data timestamp, audio data, and audio timestamp are used to generate a joint processing result of the written content image data and audio data, including:

[0019] Text recognition is performed on the image data of the written content to obtain the text recognition result; the text recognition result is associated with a data timestamp.

[0020] The audio data collected by the smart pen is recognized to obtain audio recognition results, which are associated with audio timestamps; and

[0021] Based on the associated data timestamps and audio timestamps, the text recognition results and audio recognition results are correlated to obtain the joint processing result.

[0022] In one embodiment, the method further includes:

[0023] Obtain the text search instructions and determine the search keywords from the text search instructions;

[0024] Identify the target search results that match the search keywords from the text recognition results;

[0025] Based on the target data timestamp associated with the target retrieval result, a match is made between the audio timestamps to determine the target audio timestamp corresponding to the target data timestamp; and

[0026] The corresponding audio segment is determined based on the target audio timestamp, and the audio segment is played.

[0027] In one embodiment, the method further includes:

[0028] Once the combined processing result is obtained, it is displayed on the preset display module.

[0029] In one embodiment, the method further includes:

[0030] Identify the audio segment currently in playback from the audio data;

[0031] Determine the target audio timestamp associated with the audio segment;

[0032] Based on the target audio timestamp, match it with the data timestamps to determine the target data timestamp corresponding to the target audio timestamp; and

[0033] The associated target text recognition results are determined based on the target data timestamp, and the target text recognition results are highlighted.

[0034] Thirdly, this application also provides a smart pen, which includes a camera, an audio acquisition component, a control module, and a hardware clock source. The control module is used to perform the steps of the method described above.

[0035] In one embodiment, the smart pen also includes a storage module:

[0036] The storage module is used to store the written content image data, data timestamps, audio data, and audio timestamps. The timelines of the written content image data and the audio data are aligned based on the same origin. The timeline of the written content image data is composed of data timestamps, and the timeline of the audio data is composed of audio timestamps.

[0037] In one embodiment, the camera is located at the tip of the smart pen, and the audio acquisition component is located in the middle or end of the pen body.

[0038] In one embodiment, a pressure sensor is also provided at the tip of the smart pen;

[0039] A pressure sensor is used to collect pressure signals, which indicate that a writing action has occurred.

[0040] The control module is also used to control the camera to capture image data of the written content when a pressure signal is detected.

[0041] Fourthly, this application also provides a smart terminal, which includes a display module, a memory, and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps described above to generate a joint processing result.

[0042] The display module is used to display the combined processing results.

[0043] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in any one of the above descriptions.

[0044] Sixthly, this application also provides an audio and written content joint processing system, the system including a smart terminal and a smart pen as described in any one of the above descriptions;

[0045] A smart pen for performing the steps of any of the methods described above;

[0046] A smart terminal for performing the steps of the method as described in any one of the above statements.

[0047] The aforementioned method for joint processing of audio and written content, a smart pen, and a smart terminal acquire image data of written content captured by a camera and timestamps the image data based on a hardware clock source; acquire audio data captured by an audio acquisition component and timestamps the audio data based on a hardware clock source; send the image data of written content and the corresponding timestamp, as well as the audio data and the corresponding timestamp, to the smart terminal, instructing the smart terminal to perform text recognition on the image data of written content, and associate the recognized text content with the audio data based on their respective timestamps to generate a time synchronization association between the text content and the audio data.

[0048] In this embodiment, a unified hardware clock source is integrated within the smart pen, and time stamps are applied to both image and audio data simultaneously based on this clock source. This allows for automatic association of image and audio data via timestamps. Furthermore, the smart pen completes the timestamp application at the data acquisition source, significantly reducing the computational resource consumption of the smart terminal and improving processing efficiency and response speed. Finally, the smart pen proposed in this application can write on any medium without requiring special paper. It directly captures the user's handwriting on the writing surface using a camera, without relying on any pre-printed encoding or special paper textures. Users can freely use it on any ordinary paper or writing medium, greatly reducing usage costs and lowering the barrier to entry. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0050] Figure 1 This is a flowchart illustrating a method for jointly processing audio and written content in one embodiment;

[0051] Figure 2 This is a flowchart illustrating the process of generating the joint processing result in one embodiment; and

[0052] Figure 3 A structural block diagram of an audio and written content joint processing device in one embodiment; and

[0053] Figure 4 A structural block diagram of a smart terminal in one embodiment; and

[0054] Figure 5 This is a block diagram of an audio and written content joint processing system in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] The technical background of this application will be explained below.

[0057] Several related technologies exist that can link audio recordings with handwritten notes, mainly including the following categories:

[0058] 1. Dot-matrix pen OID (Object Identifier) ​​encoding technology requires OID-encoded paper in conjunction with a smart pen. The smart pen has a built-in recognition camera that determines the writing position coordinates (X, Y) by recognizing the pre-printed OID micro-dot matrix codes on the paper, and continuously tracks position changes to reconstruct the handwriting trajectory, thereby restoring the written content. Audio and video recording are completed by an external independent device (smartphone or digital camera), and the two are linked through UTC timestamps. When reviewing, the user taps the already written abbreviated position on the paper with the pen, matches the position coordinates through the OID encoding, and finds the audio and video content corresponding to the timestamp. However, this method relies on indirect reconstruction of handwriting through OID position tracking to restore the written content. This process depends entirely on the pre-printed encoding on special OID paper. OID encoding requires custom printing, has limited supply channels, and is much more expensive than ordinary paper. Users cannot use it on ordinary white paper, notebooks, sticky notes, whiteboards, or other everyday writing media. Furthermore, the indirect recognition path of the dot matrix pen—"coordinate sampling → trajectory connection → handwriting reconstruction"—is limited in accuracy by the OID pattern density and sampling frame rate. When writing quickly, the spacing between coordinate sampling points increases, and the reconstructed handwriting may exhibit distortions such as broken strokes, jagged edges, and deformations. Position tracking can only reconstruct the pen tip's movement trajectory and cannot capture writing details such as variations in pen thickness and ink density. It also has a weak ability to reconstruct non-textual content such as graphics and tables.

[0059] 2. Traditional voice recorder technology uses a built-in microphone to capture sound, which is then converted from analog to digital and stored as a digital audio file. It can only record audio, requiring manual association between the audio data and the handwritten notes. This manual association is inefficient and inaccurate.

[0060] 3. Photo recording combined with audio recording technology, which uses a mobile phone or camera to take pictures of paper notes for archiving and links them with audio files for storage. This requires the user to manually take pictures, but like the above, it also lacks an automatic linking mechanism between notes and audio recordings.

[0061] 4. Electromagnetic induction handwriting tablet solution: This involves writing on a specially designed handwriting tablet. The tablet uses electromagnetic induction technology to detect the position of the stylus, and works with an external recording device to achieve simultaneous recording of handwriting and audio. However, this method requires a dedicated handwriting tablet and a capacitive stylus. Furthermore, the smart pen itself does not have recording capabilities, requiring users to carry and operate two separate devices simultaneously, resulting in poor portability.

[0062] In summary, the main problems with the relevant technology are: it cannot automatically link audio recordings and handwritten content, and it requires writing on a special writing medium, resulting in high costs and poor portability.

[0063] Based on this, this application proposes a method for joint processing of audio and handwritten content, applied to a smart pen. The smart pen includes a camera and an audio acquisition component, and has a built-in hardware clock source. The smart pen is communicatively connected to a smart terminal. The method includes: acquiring image data of handwritten content captured by the camera, and timestamping the image data based on the hardware clock source; acquiring audio data captured by the audio acquisition component, and timestamping the audio data based on the hardware clock source; sending the image data of handwritten content, the timestamp, the audio data, and the timestamp to the smart terminal, instructing the smart terminal to perform text recognition on the image data of handwritten content, and associating the recognized text content with the audio data based on their respective timestamps to generate a time synchronization association between the text content and the audio data. Through the embodiments of this application, the association between voice and handwritten content can be automatically realized, and writing can be performed on any writing medium without the need for a specific writing medium or carrier. See the following examples for details.

[0064] In one exemplary embodiment, such as Figure 1 As shown, a method for joint processing of audio and written content is provided, applied to a smart pen. The smart pen includes a camera and an audio acquisition component, and has a built-in hardware clock source. The smart pen communicates with a smart terminal. The method includes:

[0065] S110 acquires the image data of the written content captured by the camera, and timestamps the image data of the written content based on the hardware clock source.

[0066] The smart pen is equipped with a camera and an audio acquisition component. The camera is preferably located at the front of the smart pen and can be used to photograph and record the content written by the user. In a preferred embodiment, an ultra-small CMOS (Complementary Metal-Oxide-Semiconductor) image sensor with a resolution of 640×480 pixels or higher can be integrated at the tip of the smart pen. The sensor is equipped with a wide-angle lens with a large f / 2.0 aperture and a field of view of 120 degrees, which can cover a sufficient writing area around the pen tip.

[0067] The sensor operates at frame rates ranging from 120fps to 240fps, enabling it to capture subtle movements during rapid writing. To reduce power consumption, the sensor employs dynamic frame rate adjustment technology: it drops to 20fps when stationary or moving slowly, and automatically increases to the maximum frame rate when rapid movement is detected.

[0068] The camera module includes the following core components: a CMOS image sensor (an ultra-small, low-power CMOS sensor with a pixel size of approximately 1.4μm, supporting global shutter mode to eliminate rolling shutter effect), a miniature lens (wide-angle lens design with a focal length of approximately 0.8mm, an aperture of f / 2.0, and a field of view of 120 degrees), an infrared filter (an integrated infrared cut-off filter to avoid interference with image quality from ambient infrared light), and a flexible circuit board (using a flexible printed circuit board to connect the camera module and the main control chip, adapting to the curved structure of the pen body).

[0069] The audio acquisition component is preferably located in the middle or rear section of the smart pen for acquiring audio data. The smart pen also integrates a control chip with computing capabilities, and its shape is similar to a regular writing pen for ease of use. The aforementioned hardware clock source is an independent, high-precision timing unit on the smart pen's control chip; in one embodiment, it may include a crystal oscillator machine drive circuit, thereby providing a stable timeline for the entire smart pen system.

[0070] In this embodiment, the same main control chip can be used to assign timestamps based on the same hardware clock source to the image data and audio data of the written content. The time synchronization error can be controlled within 1ms, without the need for network connection or external calibration.

[0071] In this embodiment of the application, when the user activates the writing mode of the smart pen, the control chip inside the smart pen drives the camera to start working. The camera can take continuous pictures of the writing area in front of the pen tip at a fixed frequency (e.g., 30 frames or 60 frames per second) to obtain image data of the writing content.

[0072] Furthermore, the image data of the written content is timestamped based on a hardware clock source, thus obtaining a data timestamp. Specifically, the data timestamp refers to generating a unique time identifier for each frame or several frames of image data based on the real-time clock information provided by the hardware clock source after the camera has acquired the image of the written content, and then binding and recording this time identifier with the image data in an associative manner.

[0073] S120: Acquire audio data collected by the audio acquisition component and timestamp the audio data based on the hardware clock source.

[0074] The audio timestamp refers to the process of audio acquisition components (such as microphones, pickups, etc.) acquiring audio data. Based on the real-time clock information provided by the aforementioned hardware clock source, a corresponding acquisition time identifier is generated for a specific time unit in the audio data (such as each audio data packet, each frame of audio data, or each fixed-duration audio sample block). This time identifier is then bound and recorded with the corresponding unit of audio data in an associative manner.

[0075] In this embodiment, the audio acquisition component and the camera are started synchronously. The control chip drives the audio acquisition component to start working, and the audio acquisition component continuously acquires ambient sound and converts the ambient sound into audio data. For ease of processing, the audio data is usually divided into small blocks of fixed duration (audio frames, for example, each frame is 50-100 milliseconds long). Similarly, the start time of each audio data block is marked with a timestamp using a hardware clock source. Since the camera and the audio acquisition component share the same hardware clock source, even if there is a slight difference in their start times, their timelines are essentially the same, laying the foundation for subsequent precise alignment.

[0076] S130: The written content image data and corresponding data timestamp, audio data and corresponding audio timestamp are sent to the smart terminal, instructing the smart terminal to perform text recognition on the written content image data, and associate the recognized text content and audio data based on their respective timestamps to generate a time synchronization association between the text content and audio data.

[0077] In this embodiment, the smart pen uses a built-in wireless communication module (such as Bluetooth or Wi-Fi) to send the written content image data and audio data, each with its own timestamp, to a smart terminal in real time or in a packaged form. The smart terminal has strong computing power, a display interface, and wireless communication capabilities; it can be a smartphone, computer, or similar device.

[0078] After receiving the image data of the written content, the smart terminal uses a recognition algorithm to identify the data and thus determine the corresponding text content. Similarly, the identified text content is arranged in chronological order of writing. Ultimately, the smart terminal obtains a complete text stream, in which each character or each consecutive character is associated with one or more data timestamps.

[0079] Finally, the smart terminal matches the text content and audio data based on the data timestamp and audio timestamp. For example, it associates the text content at time a with the corresponding audio time segment [b, c] (assuming [b, c] includes a), thereby establishing the association between the text content and the audio data.

[0080] It should be noted that although this application primarily uses direct interaction between local terminals (such as pen and screen) as an example to illustrate the technical solution, this does not constitute a limitation on the implementation scenario of this application. Those skilled in the art will understand that some or all of the operations described in this application are not limited to being performed by a smart pen or smart terminal; they can also be performed collaboratively or independently by cloud servers, backend servers, edge computing nodes, software applications (Apps), web clients, or third-party service platforms that are communicatively connected to the smart pen / smart terminal. The specific task allocation among the various executing entities can be adaptively adjusted according to the actual network environment, computing resources, and business needs, all of which fall within the scope of protection covered by this application.

[0081] In this embodiment, a unified hardware clock source is integrated within the smart pen, and time stamps are applied to both image and audio data simultaneously based on this clock source. This allows for automatic association of image and audio data via timestamps. Furthermore, the smart pen completes the timestamp application at the data acquisition source, significantly reducing the computational resource consumption of the smart terminal and improving processing efficiency and response speed. Finally, the smart pen proposed in this application can write on any medium without requiring special paper. It directly captures the user's handwriting on the writing surface using a camera, without relying on any pre-printed encoding or special paper textures. Users can freely use it on any ordinary paper or writing medium, greatly reducing usage costs and lowering the barrier to entry.

[0082] In one exemplary embodiment, the image data of the written content is timestamped based on a hardware clock source, including:

[0083] Frame-by-frame recognition of the collected multi-frame image data of written content; and

[0084] If the similarity between the written content image data of the current frame and the written content image data of the previous frame is less than a preset similarity threshold, the current frame is determined as the target frame, and the written content image data corresponding to the target frame is timestamped according to the hardware clock source.

[0085] In this embodiment, the collected consecutive frames of written content image data are stored sequentially, and each frame of image data is detected. The similarity between each frame of image data and the previous frame of image data is calculated, and the similarity calculation result is compared with a preset similarity threshold. If the similarity is greater than or equal to the threshold, it means that the current target frame is highly similar to the previous frame of written content image data. This means that the user has not written any new content or has written very little new content.

[0086] If the similarity between the target frame and the previous frame is less than the threshold, it means that the user has written a certain amount of new content, resulting in a large number of pixel changes between the two frames.

[0087] When the current frame is determined to be the target frame, the main control chip immediately reads the acquisition time of the target frame according to the hardware clock source, thereby realizing the data timestamp marking of the written content image data corresponding to the target frame.

[0088] Because the writing process involves numerous instances of the pen tip remaining stationary or repeatedly drawing in the same position, the consecutive frames captured at these moments exhibit extremely high similarity and contain almost no new information. This embodiment filters out these redundant frames, only timestamping and transmitting data to target frames where the content has significantly changed. This directly reduces the number of times the controller performs timestamp reading operations and improves the overall response speed.

[0089] In one exemplary embodiment, audio data is timestamped based on a hardware clock source, including:

[0090] Multiple audio data segments are acquired based on a preset audio acquisition step size, and each audio data segment is timestamped based on a hardware clock source.

[0091] The audio acquisition step size is preset and represents the duration of each continuous audio signal acquisition by the audio acquisition component, such as 50 milliseconds or 100 milliseconds.

[0092] In this embodiment of the application, when each segment of audio data is acquired by the audio acquisition component according to the audio acquisition step size, the first frame and the last frame of the audio data segment are timestamped by the hardware clock source. For example, the first frame of the segment is the 20th millisecond and the last frame is the 70th millisecond, so that the start time and end time of the entire audio data segment can be determined. The start time and end time are the audio timestamp marks of the audio data segment.

[0093] This application also provides a method for jointly processing audio and written content, applied to a smart terminal used for communication with a smart pen, the method comprising:

[0094] The system acquires image data of the written content, data timestamp, audio data, and audio timestamp sent by the smart pen. The image data of the written content is captured by the smart pen's camera. The data timestamp is obtained by timestamping the image data of the written content based on the hardware clock source built into the smart pen. The audio data is captured by the audio acquisition component of the smart pen. The audio timestamp is obtained by timestamping the audio data based on the hardware clock source.

[0095] Based on the written content image data, data timestamp, audio data, and audio timestamp, generate the joint processing result of the written content image data and audio data.

[0096] In this embodiment of the application, after the smart pen and the smart terminal establish a wireless communication connection, the smart terminal continuously listens to the designated communication port. When the smart pen sends the above-mentioned written content image data, data timestamp, audio data and audio timestamp to the smart terminal, the smart terminal performs demodulation, decoding and other operations to restore the original data packet.

[0097] The smart terminal can call the local OCR (Optical Character Recognition) algorithm to process the image data of each frame of written content, recognize the text corresponding to each handwritten character, and associate the recognized text with the data timestamp of its source image frame, thus forming a connection and binding between the text and the data timestamp.

[0098] Similarly, audio data can be directly bound to the corresponding audio timestamp for storage, or each audio data segment can be processed by speech recognition to convert it into a transcript, and each transcript can be associated with the corresponding audio timestamp.

[0099] Finally, the image data, audio data, and their corresponding timestamps are synchronously associated. In one embodiment, the timelines of the image data and the audio data can be aligned, so that for each image data, the audio data recorded at the same time can be determined, and for each audio data segment, the image data recorded at the same time can be determined. In another embodiment, a complete set of joint processing results that integrates the image data and audio data in chronological order can be compiled based on the image data and audio data. For example, at 30s, the audio data records "It is recommended that everyone hold a meeting next week," and the handwritten data around 30s (such as at 35s) records "It would be best to hold a meeting next Wednesday." At this time, natural language processing technology can be used to compare the temporal proximity and semantic relevance of the speech-to-text and handwritten recognition text. That is, the content of the joint processing result at 30s can be "A meeting is needed next week (it would be best to hold a meeting next Wednesday)." The fusion process not only preserved the core information of the original speech but also added the specific date recorded in the handwriting later, making the record more complete and readable. Furthermore, the case also considered the timestamps of the handwritten content and the audio data during the fusion process, avoiding the fusion of data with large time spans and improving the accuracy of data fusion.

[0100] In one exemplary embodiment, such as Figure 2As shown, based on writing content image data, data timestamps, audio data and audio timestamps, a joint processing result of the writing content image data and the audio data is generated, including:

[0101] In S210, text recognition is performed on the writing content image data to obtain a text recognition result; the text recognition result is associated with a data timestamp.

[0102] Herein, the text recognition result refers to information output after the intelligent terminal performs OCR recognition on the writing content image data, and the text recognition result is still associated with a corresponding data timestamp. For example, if the text recognition result obtained by performing text recognition on the writing content image data corresponding to the timestamp at the 51st second is "Tian", then the character "Tian" is associated with the timestamp of the 51st second.

[0103] In S220, the audio data collected by the smart pen is recognized to obtain an audio recognition result, and the audio recognition result is associated with an audio timestamp.

[0104] As described above, the audio recognition result refers to data output after the intelligent terminal performs speech recognition on the audio data, and the audio recognition result is also associated with a corresponding audio timestamp.

[0105] In S230, the text recognition result and the audio recognition result are associated according to the associated data timestamp and audio timestamp to obtain a joint processing result.

[0106] In one embodiment, the time axis of the writing content image data and the time axis of the audio data may be aligned, so that each text recognition result can correspond to the audio recognition result at the same moment, and vice versa. Further, when a user views the text recognition result, the corresponding audio recognition result at the same moment can be automatically determined; similarly, when the user views the audio recognition result, the corresponding text recognition result at the same moment can also be automatically determined.

[0107] In another embodiment, the text recognition result and the audio recognition result that are close in time and related in content may also be fused. For example, if the audio recognition result is "The project deadline is Friday", the text recognition result is "March 15th", and the timestamps of the two are close, the fusion result is "The project deadline is Friday (March 15th)".

[0108] In summary, the text recognition result and the audio recognition result can be associated to obtain the joint processing result.

[0109] Through the embodiments of the present application, the recording position can be located according to the note content, and the note content can also be located according to the recording position, which effectively improves the flexibility of information retrieval. Moreover, the recorded text recognition result and audio recognition result can be fused to form a complete joint processing result with high readability.

[0110] Furthermore, this application also includes a real-time data synchronization function, that is, when the smart terminal and the smart pen are wirelessly connected, it automatically detects whether there is data stored on the smart pen that has not been synchronized with the smart terminal. If so, it incrementally synchronizes the written content image data and audio data to the smart terminal, performs text recognition and speech recognition on the incremental content, binds the text recognition content with timestamps, and the audio recognition content with timestamps, synchronizes the timeline, and establishes a full-text search index.

[0111] In one exemplary embodiment, the method further includes:

[0112] Once the combined processing result is obtained, it is displayed on the preset display module.

[0113] The display module belongs to the smart terminal, such as the screen on the smart terminal or connected to the smart terminal.

[0114] In one exemplary embodiment, the method further includes:

[0115] Obtain the text search instructions and determine the search keywords from the text search instructions;

[0116] Identify the target search results that match the search keywords from the text recognition results;

[0117] Based on the target data timestamp associated with the target retrieval results, a match is made between the audio timestamps to determine the target audio timestamp corresponding to the target data timestamp; and

[0118] The associated audio segment is determined based on the target audio timestamp, and the audio segment is played.

[0119] This application provides a method for searching and locating recordings based on text recognition content. First, the search keywords (such as "company policy") input by the user are obtained. This search instruction can be text input or voice input. Then, a full-text search and matching is performed on the text recognition results to determine the corresponding target search result. The target data timestamp of the target search result is determined (e.g., t=1500ms). Thus, t=1500ms is located on the timestamp of the text recognition result (that is, the timestamp of the written content image data).

[0120] Then, the position t=1500ms is determined on the timeline of the audio data, and the corresponding audio segment is played according to this timestamp.

[0121] In another embodiment, when the corresponding audio segment is played, the written content in the corresponding text recognition result can also be highlighted at the same time.

[0122] In one exemplary embodiment, the method further includes:

[0123] Identify the audio segment currently in playback from the audio data;

[0124] Determine the target audio timestamp associated with the audio segment;

[0125] Match the target audio timestamp with the data timestamps to determine the target data timestamp that corresponds to the target audio timestamp.

[0126] The associated target text recognition results are determined based on the target data timestamp, and the target text recognition results are highlighted.

[0127] In this embodiment of the application, a method for determining text content based on an audio segment is provided. First, the audio segment currently in playback is determined, and the current playback time is tracked in real time, i.e., the target audio timestamp mentioned above, for example, t=3000ms. Then, based on the timestamp, t=3000ms is located on the writing content timeline, and the corresponding target text recognition result is highlighted.

[0128] This application also provides a smart pen, which includes a camera, an audio acquisition component, a control module, and a hardware clock source. The control module is used to perform the steps described above.

[0129] In this embodiment, the smart pen adopts an integrated pen-shaped design for easy grip and portability. In one embodiment, the smart pen's diameter can be 150mm × 13mm, conforming to ergonomic design for comfortable holding. The total weight is approximately 25-35g (including battery), close to the weight of a regular fountain pen, facilitating prolonged use. In practical applications, aluminum alloy or high-performance plastic materials can be used, balancing lightweight and durability. Color schemes can be designed by the user.

[0130] The pen cap features a USB charging port, a power button, and an indicator light. Pressing the power button simultaneously activates voice recording and handwriting capture, while the indicator light shows whether these functions are currently enabled.

[0131] A battery module (300mAh, 3.7V) is located at the rear of the pen body, and a storage module, control module and power management chip can also be installed.

[0132] A microphone module can be installed in the middle section (slightly above) of the pen body for 360° sound pickup, facilitating sound capture. A communication module can also be installed in the middle section of the pen body.

[0133] The pen tip can be equipped with a camera module, CMOS sensor, infrared LED, and pressure sensor. The very tip is used to capture the written content.

[0134] Through the embodiments of this application, the pen tip adopts a spring structure, balancing pressure sensing and writing feel to ensure smooth and natural writing; furthermore, this smart pen is compatible with standard D1 or D3 ink cartridges, which users can replace themselves to meet long-term use needs. Furthermore, the pen tip module of this smart pen is detachable and replaceable, facilitating maintenance and repair. Finally, the battery and control module are located in the rear middle section of the pen body, with a center of gravity design similar to that of a regular ballpoint pen, making it suitable for extended writing sessions.

[0135] In one exemplary embodiment, the smart pen further includes a storage module:

[0136] The storage module is used to store the written content image data, data timestamps, audio data, and audio timestamps. The timelines of the written content image data and the audio data are aligned based on the same origin. The timeline of the written content image data is composed of data timestamps, and the timeline of the audio data is composed of audio timestamps.

[0137] In this application, when the user presses the record button on the pen cap, the main control chip simultaneously activates the writing content acquisition module (camera) and the recording module (microphone), ensuring that both start working at the same moment. Furthermore, the main control chip assigns timestamps based on the same clock source to the writing content image data (the acquisition time of each frame) and the audio data (the start and end times of each audio segment).

[0138] Similarly, during data storage, the timeline of written content and the timeline of audio recording are aligned based on the same origin, ensuring that the written content at any given moment can accurately correspond to the corresponding audio segment. The timestamp accuracy of this application reaches the millisecond level, meeting the precise correspondence requirements of scenarios such as meeting minutes and classroom notes.

[0139] For example, at time T0, the writing content acquisition module and the recording module are started synchronously. The timestamps of the writing content images are: Frame 1 [t=0ms], Frame 2 [t=8ms], Frame 3 [t=17ms], Frame 4 [t=25ms], etc.; the timestamps of the audio data are: Audio segment 1 [t=0ms, duration=100ms], Audio segment 2 [t=100ms, duration=100ms], Audio segment 3 [t=200ms, duration=100ms], etc.

[0140] At time T1, the search is performed by recognizing the text content. For example, if the keyword "company policy" is searched, the text recognition results will find t=150ms. Based on the time axis, the audio segment 2 (t=100~200ms) will be located and the corresponding recording segment will be played.

[0141] Similarly, at time T2, the handwriting recognition result can also be located through audio data. For example, by playing audio at t=300ms, the writing content at the corresponding time can be determined according to the timeline and highlighted.

[0142] In one exemplary embodiment, the camera is located at the tip of the smart pen, and the audio acquisition component is located in the middle or end of the pen body.

[0143] It should be noted that the above location descriptions are for illustrative purposes only. In actual applications, the camera, audio acquisition components, control module, hardware clock source, and other hardware units can be configured on the smart pen according to actual needs.

[0144] In one exemplary embodiment, a pressure sensor is also provided at the tip of the smart pen;

[0145] A pressure sensor is used to collect pressure signals, which indicate that a writing action has occurred.

[0146] The control module is also used to control the camera to capture image data of the written content when a pressure signal is detected.

[0147] In another embodiment, the location of the pressure sensor can be set by relevant technicians according to actual needs, for example, it can be set at the front of the smart pen or at the position where the hand grips it.

[0148] In summary, in this embodiment of the application, the written content is directly captured by a camera, and a deep learning OCR model is used for text recognition, so that the physical written content is directly converted into searchable and editable digital text, which can be used on any ordinary writing medium.

[0149] Furthermore, this application integrates the camera and audio acquisition module into the same pen body, with all modules sharing the same main control chip and the same hardware clock source. Users only need to carry one pen to simultaneously complete the acquisition of written content and the recording of audio, truly realizing an integrated portable design.

[0150] Because the audio data and the handwritten image data are timestamped using the same clock source, they can be linked through timestamps. Users can directly search the entire handwritten notes by entering keywords to quickly locate the corresponding audio segment, representing a significant leap in efficiency compared to physically clicking on the paper. Furthermore, digital content exists independently of physical paper; even if the paper is lost or not readily available, users can still remotely search and retrieve the information via the app.

[0151] Furthermore, this application, by directly converting handwritten content into digital text through OCR recognition on a smart terminal, enables intelligent processing of the notes, including keyword extraction, semantic analysis, summary generation, and multilingual translation. It also supports exporting audio recordings and notes to PDF and Word formats, significantly improving the usability and value of the records. Compared to the recognition method of indirectly reconstructing handwriting through OID position tracking using a dot matrix pen, the direct OCR recognition path of this invention is more efficient and accurate, realizing a complete conversion chain from physical writing to intelligent digital content.

[0152] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0153] Based on the same inventive concept, this application also provides an audio and written content joint processing apparatus for implementing the aforementioned audio and written content joint processing method. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations of one or more audio and written content joint processing apparatus embodiments provided below can be found in the limitations of the audio and written content joint processing method described above, and will not be repeated here.

[0154] In one exemplary embodiment, such as Figure 3 As shown, an audio and written content co-processing device is provided, comprising:

[0155] The acquisition module 31 is used to acquire the writing content image data, data timestamp, audio data, and audio timestamp sent by the smart pen; wherein, the writing content image data is acquired by the smart pen's camera; the data timestamp is obtained by timestamping the writing content image data based on the hardware clock source built into the smart pen; the audio data is acquired by the smart pen's audio acquisition component; and the audio timestamp is obtained by timestamping the audio data based on the hardware clock source.

[0156] The joint processing module 32 is used to generate a joint processing result of the written content image data and audio data based on the written content image data, data timestamp, audio data and audio timestamp.

[0157] Each module in the aforementioned audio and written content joint processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0158] In one exemplary embodiment, such as Figure 4 As shown, a smart terminal is provided, which includes a display module 41, a memory 42 and a processor 43. The memory 42 stores a computer program, and the processor 43 executes the computer program to implement the steps of the method as described in any one of the above descriptions and generate a joint processing result.

[0159] Display module 41 is used to display the combined processing results.

[0160] In one exemplary embodiment, such as Figure 5 As shown, an audio and written content joint processing system is provided, the system including a smart terminal 52 and a smart pen 51 as described in any one of the above descriptions;

[0161] Smart pen 51, used to perform the steps of the method as described in any one of the above statements;

[0162] The smart terminal 52 is used to perform the steps of the method as described in any one of the above descriptions.

[0163] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the audio and written content joint processing method as described in any of the preceding embodiments.

[0164] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the audio and written content joint processing method as described in any of the preceding embodiments.

[0165] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the audio and written content joint processing method as described in any of the preceding embodiments.

[0166] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0167] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0168] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for jointly processing audio and written content, characterized in that, The method is applied to a smart pen, which includes a camera and an audio acquisition component, and has a built-in hardware clock source. The smart pen is communicatively connected to a smart terminal. The image data of the written content captured by the camera is acquired, and the image data of the written content is timestamped based on the hardware clock source; Acquire audio data collected by the audio acquisition component, and timestamp the audio data based on the hardware clock source; and The written content image data and the corresponding data timestamp, the audio data and the corresponding audio timestamp are sent to the smart terminal, instructing the smart terminal to perform text recognition on the written content image data, and associate the recognized text content with the audio data based on their respective timestamps to generate a time synchronization association between the text content and the audio data.

2. The method according to claim 1, characterized in that, The process of timestamping the written content image data based on the hardware clock source includes: Frame-by-frame recognition of the collected multi-frame image data of written content; and If the similarity between the written content image data of the current frame and the written content image data of the previous frame is less than a preset similarity threshold, the current frame is determined as the target frame, and the written content image data corresponding to the target frame is timestamped according to the hardware clock source.

3. The method according to claim 1 or 2, characterized in that, The step of timestamping the audio data based on the hardware clock source includes: Multiple audio data segments are acquired according to the preset audio acquisition step size, and each audio data segment is timestamped based on the hardware clock source.

4. A method for jointly processing audio and written content, characterized in that, Applied to a smart terminal, wherein the smart terminal is communicatively connected to a smart pen, the method includes: The system acquires image data of the written content, a data timestamp, audio data, and an audio timestamp sent by the smart pen. The image data of the written content is captured by the smart pen's camera. The data timestamp is obtained by timestamping the image data of the written content using a hardware clock source built into the smart pen. The audio data is captured by the smart pen's audio acquisition component. The audio timestamp is obtained by timestamping the audio data using the hardware clock source. Based on the written content image data, the data timestamp, the audio data, and the audio timestamp, a joint processing result of the written content image data and the audio data is generated.

5. The method according to claim 4, characterized in that, The written content image data, the data timestamp, the audio data, and the audio timestamp are used to generate a joint processing result of the written content image data and the audio data, including: The image data of the written content is subjected to text recognition to obtain a text recognition result; the text recognition result is associated with the data timestamp. The audio data collected by the smart pen is recognized to obtain an audio recognition result, which is associated with the audio timestamp; and The text recognition result and the audio recognition result are associated with the associated data timestamp and the audio timestamp to obtain the joint processing result.

6. The method according to claim 5, characterized in that, Also includes: Obtain a text search instruction, and determine search keywords from the text search instruction; Determine the target search result that matches the search keyword from the text recognition results; Based on the target data timestamp associated with the target retrieval result, a match is made in the audio timestamp to determine the target audio timestamp corresponding to the target data timestamp; as well as The corresponding audio segment is determined based on the target audio timestamp, and the audio segment is played.

7. The method according to claim 4 or 5, characterized in that, Also includes: Upon obtaining the combined processing result, the combined processing result is displayed on a preset display module.

8. The method according to claim 4 or 5, characterized in that, Also includes: Identify the audio segment currently in playback from the audio data; Determine the target audio timestamp associated with the audio segment; The target data timestamp is determined by matching the target audio timestamp with the data timestamps. as well as The associated target text recognition result is determined based on the target data timestamp, and the target text recognition result is highlighted.

9. A smart pen, characterized in that, The smart pen includes a camera, an audio acquisition component, a control module, and a hardware clock source, wherein the control module is used to execute the method according to any one of claims 1 to 3.

10. The smart pen according to claim 9, characterized in that, The smart pen also includes a storage module: The storage module is used to store the written content image data, data timestamps, audio data, and audio timestamps. The timelines of the written content image data and the audio data are aligned based on the same origin. The timeline of the written content image data is composed of the data timestamps, and the timeline of the audio data is composed of the audio timestamps.

11. The smart pen according to claim 9, characterized in that, The camera is located at the tip of the smart pen, and the audio acquisition component is located in the middle or end of the pen body.

12. The smart pen according to claim 9, characterized in that, The smart pen is also equipped with a pressure sensor at the tip; The pressure sensor is used to collect pressure signals, which indicate that a writing action has occurred. The control module is also used to control the camera to acquire image data of the written content when a pressure signal is detected.

13. A smart terminal, characterized in that, The smart terminal includes a display module, a memory, and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method according to any one of claims 4 to 8 and generates a joint processing result. The display module is used to display the combined processing results.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 4 to 8.

15. A system for jointly processing audio and written content, characterized in that, The system includes a smart terminal and a smart pen as described in any one of claims 9 to 12; The smart pen is used to perform the steps of the method as described in any one of claims 1 to 3; The smart terminal is used to perform the steps of the method as described in any one of claims 4 to 8.

Images