A content generation method, a writing pen, and a computer readable storage medium

Abstract

The application relates to a content generation method, a writing pen and a computer readable storage medium. The method comprises the following steps: receiving at least one frame of writing image collected by a writing pen in a writing process; determining a global writing image of the writing pen in the writing process according to the at least one frame of writing image; determining a writing track of the writing pen in the writing process and pixel coordinate data of each track point on the writing track according to the global writing image; determining a connection relationship between each track point according to a writing sequence of each track point; and generating target text content according to the pixel coordinate data of each track point on the writing track and the connection relationship between each track point. The method can accurately restore complete handwriting of the writing process by determining the pixel coordinate data of each track point on the writing track and the connection relationship between each track point, so that accurate target text content can be obtained, and effective processing of the writing content can be realized.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a content generation method, a writing pen, and a computer-readable storage medium. Background Technology

[0002] With the development of information technology, people's demand for electronic management of information is growing.

[0003] However, in scenarios such as meeting minutes, classroom notes, and interviews and surveys, a large amount of information is still recorded in the traditional form of pen and paper. This method of recording information through traditional pen and paper cannot achieve the processing of quickly organizing written information, identifying key information, and summarizing and generalizing.

[0004] Therefore, how to effectively process written information is a problem worthy of attention. Summary of the Invention

[0005] Therefore, it is necessary to provide a content generation method, a writing pen, and a computer-readable storage medium that can effectively process written information, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a content generation method applied to a control terminal, comprising:

[0007] Receive at least one frame of writing image captured by the writing pen during the writing process;

[0008] Determine the global writing image of the writing pen during the writing process based on at least one frame of writing image;

[0009] Based on the global writing image, determine the writing trajectory of the writing pen during the writing process, as well as the pixel coordinate data of each trajectory point on the writing trajectory;

[0010] Determine the connection relationships between the trajectory points based on the writing order of each trajectory point; and

[0011] The target text content is generated based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point.

[0012] In one embodiment, determining the global writing image of the writing pen during the writing process based on at least one writing image frame includes: performing feature point matching on any two adjacent writing images to obtain feature point pairs in the two writing images; determining the coordinate transformation relationship between the two writing images based on the feature point pairs; performing coordinate transformation on the two writing images based on the coordinate transformation relationship to obtain the target writing images corresponding to each of the two writing images in the same coordinate space; and performing fusion processing on the obtained target writing images to obtain the global writing image of the writing pen during the writing process.

[0013] In one embodiment, determining the writing trajectory of the writing pen during the writing process and the pixel coordinate data of each trajectory point on the writing trajectory based on the global writing image includes: performing image recognition on the global writing image to obtain the writing trajectory of the writing pen during the writing process; performing binarization processing on the global writing image to obtain a binarized writing image; and extracting the pixel coordinate data of each trajectory point on the writing trajectory from the binarized writing image.

[0014] In one embodiment, generating target text content based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between the trajectory points includes: acquiring audio text content; wherein the audio text content is obtained by converting audio data collected by the writing pen during the writing process into text; and generating target text content based on the audio text content, the pixel coordinate data of each trajectory point, and the connection relationship between the trajectory pixels.

[0015] In one embodiment, obtaining audio text content includes: receiving audio data collected by a writing pen during the writing process; and converting the audio data into text to obtain audio text content.

[0016] In one embodiment, generating target text content based on audio text content, pixel coordinate data of each trajectory point, and connection relationships between trajectory pixels includes: obtaining a target document type; and generating target text content that matches the target text type based on audio text content, pixel coordinate data of each trajectory point, and connection relationships between trajectory pixels.

[0017] Secondly, this application provides a content generation method applied to a writing pen, including:

[0018] In response to the writing signal from the writing pen, acquire at least one frame of writing image captured by the writing pen during the writing process;

[0019] Send at least one frame of the written image to the control terminal;

[0020] Among them, at least one frame of writing image is used by the control terminal to determine the global writing image of the writing pen during the writing process; based on the global writing image, the writing trajectory of the writing pen during the writing process, as well as the pixel coordinate data of each trajectory point on the writing trajectory, are determined; based on the writing order of each trajectory point, the connection relationship between each trajectory point is determined; and based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point, the target text content is generated.

[0021] In one embodiment, the content generation method further includes:

[0022] Acquire audio data collected by the writing pen during the writing process;

[0023] The audio data is sent to the control terminal.

[0024] Thirdly, this application also provides a writing pen, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0025] In response to the writing signal from the writing pen, acquire at least one frame of writing image captured by the writing pen during the writing process;

[0026] Send at least one frame of the written image to the control terminal;

[0027] Among them, at least one frame of writing image is used by the control terminal to determine the global writing image of the writing pen during the writing process; based on the global writing image, the writing trajectory of the writing pen during the writing process, as well as the pixel coordinate data of each trajectory point on the writing trajectory, are determined; based on the writing order of each trajectory point, the connection relationship between each trajectory point is determined; and based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point, the target text content is generated.

[0028] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the content generation method of any of the above aspects.

[0029] In the above content generation method, the control terminal can receive at least one frame of writing image captured by the writing pen during the writing process; determine the global writing image of the writing pen during the writing process based on the at least one frame of writing image; determine the writing trajectory of the writing pen during the writing process and the pixel coordinate data of each trajectory point on the writing trajectory based on the global writing image; determine the connection relationship between each trajectory point based on the writing order of each trajectory point; and generate the target text content based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point. The above content generation method determines the global writing image of the writing pen during the writing process based on at least one frame of writing image, thus obtaining the complete writing content of the writing process, laying the foundation for obtaining accurate target text content. In addition, by determining the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point, the complete handwriting of the writing process can be accurately restored, thereby obtaining accurate target text content and realizing effective processing of the writing content. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0031] Figure 1 This is a diagram illustrating the application environment of a content generation method in one embodiment.

[0032] Figure 2 This is a flowchart illustrating a content generation method in one embodiment;

[0033] Figure 3 This is a schematic diagram illustrating the fusion of a writing image in one embodiment;

[0034] Figure 4 This is a schematic diagram of a binarized written image in one embodiment;

[0035] Figure 5 This is a schematic diagram of the pixel coordinate data of each trajectory point in one embodiment;

[0036] Figure 6 This is a schematic diagram illustrating the connection relationships between trajectory points in one embodiment;

[0037] Figure 7 This is a flowchart illustrating the content generation method in another embodiment;

[0038] Figure 8 This is a flowchart illustrating the content generation method in yet another embodiment;

[0039] Figure 9 This is a block diagram of the internal structure of a stylus in one embodiment;

[0040] Figure 10 This is a structural block diagram of a content generation device in one embodiment;

[0041] Figure 11 A structural block diagram of the content generation apparatus in yet another embodiment; and

[0042] Figure 12 This is a diagram of the internal structure of the stylus in another embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] The content generation method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, the control terminal 102, server 104, and writing pen 106 can communicate via a network. A data storage system can store the data that the server 104 needs to process. The data storage system can be integrated onto the server 104 or placed on a cloud or other network server. The control terminal 102 can receive at least one frame of writing image captured by the writing pen 106 during the writing process; based on the at least one frame of writing image, determine the global writing image of the writing pen 106 during the writing process; based on the global writing image, determine the writing trajectory of the writing pen 106 during the writing process, and the pixel coordinate data of each trajectory point on the writing trajectory; determine the connection relationship between each trajectory point based on the writing order of each trajectory point; and generate the target text content based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point. The control terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. The target application is deployed on the control terminal 102. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The writing pen 106 can have image acquisition function and also recording function.

[0045] In one exemplary embodiment, such as Figure 2 As shown, a content generation method is provided, which can be applied to... Figure 1 Taking the control terminal in the example, the following steps are included:

[0046] S210 receives at least one frame of writing image captured by the writing pen during the writing process.

[0047] The writing image can be understood as the image of what is written by the writing pen.

[0048] The writing process can be understood as the period from when the writing pen detects a writing signal until the time when the writing signal can no longer be detected exceeds a preset duration. For example, the period from when the writing pen detects a writing signal until the time when the writing signal can no longer be detected can be defined as a writing process.

[0049] In some embodiments, the control terminal can receive writing images captured and sent frame by frame by the writing pen, and the control terminal can also receive multiple writing images sent by the writing pen after capturing multiple writing images.

[0050] In some embodiments, the writing image may be acquired after the writing pen detects a writing signal during the writing process. The writing signal can be understood as the signal indicating that the writing pen has begun writing. In some embodiments, the writing signal may be the pressure signal generated by the writing pen when it contacts the writing surface.

[0051] S220, determine the global writing image of the writing pen during the writing process based on at least one frame of writing image.

[0052] The global writing image can be understood as an image containing the complete writing content written by the writing pen during the writing process.

[0053] In some embodiments, at least one frame of the writing image can be stitched together to obtain a global writing image.

[0054] In some embodiments, for each of any two adjacent frames of written images, feature points in that frame can be extracted using a feature point detection algorithm. For example, the feature point detection algorithm can be at least one of the following: Scale-Invariant Feature Transform (SIFT) detection algorithm, Speeded-Up Robust Features (SURF) detection algorithm, Features from Accelerated Segment Test (FAST) and Oriented FAST and Rotated BRIEF (ORB) detection algorithms.

[0055] In some embodiments, feature point matching can be performed on any two adjacent writing images to obtain feature point pairs in the two writing images. Based on the feature point pairs, the coordinate transformation relationship between the two writing images is determined. According to the coordinate transformation relationship, the two writing images are transformed to obtain their respective target writing images in the same coordinate space. The obtained target writing images are then fused to obtain the global writing image of the writing pen during the writing process.

[0056] Feature point matching can be understood as the process of finding corresponding points at the same location in two frames of written images.

[0057] The feature point pair includes two feature points at the same location in two frames of the written image.

[0058] The coordinate transformation relationship can be understood as the transformation relationship between the coordinate systems corresponding to the two frames of the written image. The coordinate transformation relationship can include the position translation amount and / or rotation angle.

[0059] The same coordinate space can be understood as the coordinate space under the same image coordinates.

[0060] For example, the coordinates of the previous frame of the writing image can be transformed to the coordinates of the next frame of the writing image, based on the coordinate transformation relationship. The target writing image corresponding to the previous frame of the writing image is the image obtained after coordinate transformation, and the target writing image corresponding to the next frame of the writing image can be the next frame of the writing image itself.

[0061] Alternatively, the coordinates of the subsequent writing image can be transformed to the coordinates of the previous writing image based on coordinate transformation relationships. The target writing image corresponding to the subsequent writing image is the image obtained after coordinate transformation, while the target writing image corresponding to the previous writing image can be the previous writing image itself.

[0062] The fusion process can be understood as the process of merging overlapping areas of the target written image.

[0063] After performing coordinate transformation on any two adjacent writing images, the target writing images corresponding to each frame in the same coordinate space can be obtained. The obtained target writing images are then fused, specifically the overlapping areas, to obtain the global writing image of the writing pen during the writing process.

[0064] Figure 3 A schematic diagram of merging written images is shown. Taking the merging of two adjacent written images as an example, in the first frame, the first line contains the text "Hello W" and the second line contains the text "ABC"; in the second frame, the first line contains the text "o World" and the second line contains the text "DEF". In the merged global written image, the first line contains "Hello World" and the second line contains "ABC DEF". It can be seen that the global written image contains the complete content of both written images.

[0065] In some embodiments, after receiving at least one frame of writing image, the writing content area in the writing image can also be identified by an image recognition algorithm to remove the background area, and the global writing image of the writing pen during the writing process can be determined based on the writing content area.

[0066] S230, based on the global writing image, determines the writing trajectory of the writing pen during the writing process, as well as the pixel coordinate data of each trajectory point on the writing trajectory.

[0067] The writing trajectory can be understood as the trajectory formed by the writing pen at various points during the writing process.

[0068] The trajectory point can be understood as the position point of the writing pen during the writing process.

[0069] Pixel coordinate data can be understood as the coordinate data of the pixels corresponding to the location points in the written image.

[0070] In some embodiments, image recognition can be performed on the global writing image to obtain the writing trajectory of the writing pen during the writing process. In specific implementations, optical character recognition (OCR) technology can be used to perform image recognition on the global writing image to obtain the writing trajectory of the writing pen during the writing process.

[0071] Then, the global writing image can be binarized to obtain a binarized writing image, such as... Figure 4 As shown; and, from the binarized writing image, extract the pixel coordinate data of each trajectory point on the writing trajectory, such as... Figure 5 As shown.

[0072] Binarization can be understood as the process of converting a grayscale image into a binary image. During binarization, pixels with grayscale values ​​greater than or equal to a preset grayscale value are set as grayscale maxima, and pixels with preset grayscale values ​​are set as grayscale minima.

[0073] By binarizing the written image, the writing trajectory points and background pixels can be separated. In some embodiments, pixel coordinate data of each trajectory point can be extracted along the writing trajectory.

[0074] S240, determine the connection relationship between each trajectory point according to the writing order of each trajectory point.

[0075] The writing order can be understood as the order of timestamps between trajectory points and the spatial positional relationship between the written images.

[0076] In some embodiments, the trajectory points can be connected by line segments according to the connection relationship between them, such as... Figure 6 As shown, based on the connections between the trajectory points, Scalable Vector Graphics (SVG) format data, i.e., vector trajectory maps, can be generated. A vector trajectory map includes the pixel coordinates of each trajectory point and the connections between them. Vector trajectory maps are small in file size, can be scaled without loss, can accurately reproduce the original handwriting shape on any device, and support subsequent editing and format conversion.

[0077] S250 generates the target text content based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point.

[0078] The target text content can be understood as text content of a target text type. For example, the target text type can be at least one of the following: meeting minutes, study notes, research interviews, class summaries, and medical case records. Meeting minutes can be used to extract key points, speech summaries, and to-do items; study notes can be used to organize knowledge points, highlight key points, and generate mind map structures; research interviews can be used to organize interview questions and answers and extract key information; class summaries can be used to summarize course key points and generate review outlines; and medical case records can be used to organize consultation records and generate structured medical record information.

[0079] In some embodiments, in order to make the generated target text content more accurate, audio text content can also be obtained; wherein, the audio text content is obtained by converting the audio data collected by the writing pen during the writing process into text; the target text content is generated based on the audio text content, the pixel coordinate data of each trajectory point and the connection relationship between each trajectory pixel point.

[0080] In some embodiments, obtaining audio text content may include: receiving audio data collected by a writing pen during the writing process; and converting the audio data into text to obtain audio text content.

[0081] In some embodiments, the writing images and audio data collected by the writing pen are marked with timestamps. Before generating the target text content based on the audio text content, the pixel coordinate data of each trajectory point, and the connection relationship between each trajectory point, the audio text content and the pixel coordinate data of each trajectory point can also be timestamp aligned.

[0082] In some embodiments, the target document type can be obtained; and target text content matching the target text type can be generated based on the audio text content, the pixel coordinate data of each trajectory point, and the connection relationship between each trajectory pixel point.

[0083] The target text type can be selected by the user on the control terminal. For example, the target text type can include at least one of the following: meeting minutes, study notes, survey interviews, and class summaries.

[0084] The above content generation method determines the global writing image of the writing pen during the writing process based on at least one frame of writing image. In this way, the complete writing content of the writing process can be obtained, laying the foundation for obtaining accurate target text content in the future. In addition, by determining the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point, the complete handwriting of the writing process can be accurately restored, thereby obtaining accurate target text content and realizing effective processing of the writing content.

[0085] It should be noted that although this application primarily uses the direct interaction between the control terminal and the writing pen as an example to illustrate the technical solution, this does not constitute a limitation on the scenarios of the embodiments of this application. Those skilled in the art will understand that some or all of the operations described in this application are not limited to being performed by the writing pen or the control terminal; they can also be performed collaboratively or independently by cloud servers, backend servers, edge computing nodes, software applications (APPs), web clients, or third-party service platforms connected to the writing pen / control terminal. The specific task allocation among the various executing entities can be adaptively adjusted according to the actual network environment, computing resources, and business needs, all of which fall within the scope of protection covered by this application.

[0086] In one exemplary embodiment, such as Figure 7 As shown, a content generation method is provided, which can be applied to... Figure 1 Taking the writing pen as an example, the steps are as follows:

[0087] S710, in response to the writing signal of the writing pen, acquires at least one frame of writing image captured by the writing pen during the writing process.

[0088] The writing signal can be understood as the signal that the writing pen begins to write.

[0089] In some embodiments, the writing signal can be a pressure signal generated when the writing pen contacts the writing surface. In some embodiments, the pressure signal generated when the writing pen contacts the writing surface can be detected by a pressure sensor in the writing pen.

[0090] The S720 sends at least one frame of the writing image to the control terminal.

[0091] Among them, at least one frame of writing image is used for the control terminal to determine the global writing image of the writing pen in the writing process; based on the global writing image, the writing trajectory of the writing pen in the writing process and the pixel coordinate data of each trajectory point on the writing trajectory are determined; based on the writing order of each trajectory point, the connection relationship between each trajectory point is determined; and based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point, the target text content is generated.

[0092] In some embodiments, the writing pen can capture writing images frame by frame and send them to the control terminal. In some embodiments, the writing pen can capture multiple frames of writing images and then send multiple frames of writing images to the control terminal.

[0093] In some embodiments, the writing pen can also acquire audio data collected during the writing process and send the audio data to the control terminal.

[0094] The audio data sent to the control terminal can be used by the control terminal to convert the audio data into text and then generate the target text content based on the audio text content, the pixel coordinate data of each trajectory point, and the connection relationship between each trajectory point.

[0095] In some embodiments, the control terminal can decode the received audio data to obtain a Pulse Code Modulation (PCM) audio stream. Then, speech activity detection is performed on the PCM audio stream to distinguish between speech segments and non-speech segments, thereby identifying speech segments. These speech segments are then converted to text, punctuation is added, and paragraphs are created to obtain the audio text content. Semantic analysis is performed on the audio text content to extract semantic information such as keywords, entities, and topics, yielding key semantic content. Based on this key semantic content, the pixel coordinates of each trajectory point, and the connection relationships between these trajectory points, the target text content is generated.

[0096] Based on the above embodiments, it is possible to supplement missing information in a written image using audio text content, or to supplement deficiencies in audio text content using charts and key data in the written image.

[0097] In one exemplary embodiment, such as Figure 8 As shown, a content generation method is provided, including the following steps:

[0098] S801, in response to the writing signal of the writing pen, the writing pen acquires at least one frame of writing image and audio data collected during the writing process.

[0099] S802, the writing pen sends at least one frame of writing image to the control terminal, and sends audio data to the control terminal and the server.

[0100] S803, the control terminal performs feature point matching on any two adjacent written images to obtain feature point pairs in the two written images.

[0101] S804, the control terminal determines the coordinate transformation relationship between two frames of written images based on the feature point pairs.

[0102] S805, the control terminal performs coordinate transformation on the two writing images according to the coordinate transformation relationship, and obtains the target writing images corresponding to the two writing images in the same coordinate space.

[0103] S806, the control terminal performs fusion processing on the obtained target writing image to obtain a global writing image of the writing pen during the writing process.

[0104] S807, the control terminal performs image recognition on the global writing image to obtain the writing trajectory of the writing pen during the writing process.

[0105] S808, the control terminal performs binarization processing on the global writing image to obtain a binarized writing image.

[0106] S809, the control terminal extracts the pixel coordinate data of each trajectory point on the writing trajectory from the binary writing image.

[0107] S810, the control terminal determines the connection relationship between each trajectory point according to the writing order of each trajectory point.

[0108] In some embodiments, the control terminal can generate a vector trajectory map based on the connection relationship between each trajectory point and the pixel coordinate data of each trajectory point.

[0109] In some embodiments, the control terminal can perform text conversion on the audio data to obtain audio text data.

[0110] In some embodiments, the control terminal can display at least one frame of writing image, global writing image, audio text data, and audio data through a deployed target application.

[0111] S811, the control terminal sends the pixel coordinate data of each trajectory point and the connection relationship between each trajectory point to the server.

[0112] In some embodiments, the control terminal can store relevant technologies from the writing process in a knowledge base, including session metadata, raw data (audio recordings, writing images, etc.), processing results (writing trajectory, OCR text, audio text content, target text content, etc.), and semantic indexes (keywords, entities, topic vectors, time indexes, etc.). The knowledge base supports multiple retrieval methods: full-text retrieval based on OCR text and speech-to-text, semantic retrieval based on semantic vectors, timeline retrieval by time dimension, and tag retrieval by topic tags. The system automatically discovers topic associations between different knowledge items through semantic similarity, forming a knowledge network, and automatically recommends relevant historical records when the user creates a new item.

[0113] S812, the server performs text conversion on the audio data to obtain the audio text content.

[0114] S813, the server generates the target text content based on the audio text content, the pixel coordinate data of each trajectory point, and the connection relationship between the trajectory pixels.

[0115] In some embodiments, the server may use artificial intelligence (AI) summary service functions to generate target text content.

[0116] In some embodiments, the server can return the target text content to the control terminal and display the target text content through the deployed target application.

[0117] In one exemplary embodiment, a writing pen is also provided, such as Figure 9 As shown, it includes: a writing pen body, an image acquisition module, an audio acquisition module, a writing signal detection module, a main control chip, a storage module, a wireless communication module, a battery module, and a charging module.

[0118] The writing pen body is similar in appearance to traditional writing tools, and a replaceable writing pen refill is installed inside the writing pen body. At least one functional module is integrated inside the writing pen body.

[0119] The image acquisition module is located within the writing pen body near the pen tip. The image acquisition module includes a camera facing the writing surface to capture writing images. For example, the image acquisition module can be positioned on the side of the pen barrel, 2-3 cm behind the pen tip, with the camera facing the writing surface. The image acquisition module uses a wide-angle, distortion-resistant lens, with the optical field of view covering a sufficiently large writing area around the pen tip. The camera resolution is no less than 1 megapixel, supporting continuous acquisition at a minimum acquisition frequency of 3 frames per second. The lens undergoes distortion-resistant optical design to ensure minimal distortion in the edge areas of the acquired writing image, providing high-quality raw data for subsequent image stitching. In some embodiments, the acquisition frequency of the image acquisition module can be adjusted according to the writing speed; for example, the acquisition frequency can be increased during fast writing to ensure the integrity of the writing trajectory, while the acquisition frequency can be decreased during slow writing or pauses to save power.

[0120] An audio acquisition module is positioned on the writing pen body away from the area covered by the user's hand to collect audio data during the writing process. In some embodiments, the audio acquisition module can be started or stopped synchronously with the image acquisition module, or it can be started independently based on user operation to collect only audio data.

[0121] A writing signal detection module is located within the writing pen body near the pen tip and is used to detect the writing signal of the writing pen. In some embodiments, the writing signal detection module may include a pressure sensor for detecting pressure data when the writing pen contacts the writing surface. For example, the pressure sensor is mounted on the pen tip base of the writing pen body; when the pen tip contacts the writing surface and generates pressure, the pressure sensor outputs a trigger signal to the main control chip. For example, when the pressure sensor detects that the pen tip has been away from the writing surface for more than a preset time threshold (e.g., 5 seconds), the main control chip determines that writing has paused and controls the image acquisition module to stop capturing images. The audio acquisition module can continue recording or pause synchronously according to user settings.

[0122] The main control chip is connected to the image acquisition module, audio acquisition module, and writing signal detection module to coordinate the operation of each module. For example, the main control chip is located in the middle of the writing pen body and is connected to the image acquisition module, audio acquisition module, writing signal detection module, storage module, and wireless communication module via an internal bus. The main control chip can start or stop the acquisition of writing image and audio data based on the pressure signal received from the pressure sensor; control the shooting parameters of the image acquisition module; compress and perform preliminary formatting processing on the acquired writing images; manage data read / write operations in the storage module; and control data transmission in the wireless communication module.

[0123] The storage module is used to store the acquired handwriting image and audio data. For example, the storage module is located near the main control chip, uses flash memory chips, and has a capacity of no less than 64GB, for storing the acquired handwriting image and audio data. The storage module can be used to query the corresponding handwriting image and audio data based on timestamps.

[0124] The wireless communication module is used for data transmission with the control terminal, supporting at least one of short-range wireless communication and wireless local area network communication. In some embodiments, the wireless communication module can be a Bluetooth communication module and / or a Wireless Fidelity (WiFi) module. The Bluetooth communication module is integrated into the main control chip or a separate chip, supports the Bluetooth Low Energy protocol, and is used for wireless transmission with the control terminal. The WiFi module is used when large files need to be transmitted (such as a large amount of handwritten images or long-duration audio data). The main control chip controls the WiFi module to turn on, the writing pen acts as a WiFi access point, and the control terminal connects to this hotspot for high-speed point-to-point data transmission.

[0125] The battery module powers each module. For example, the battery module uses a lithium-ion rechargeable battery with a capacity of not less than 250mAh and is located in the middle and rear of the pen barrel.

[0126] The charging module is used for wired charging of the writing pen and / or charging of the charging case. A wired charging port is located at the end or side of the writing pen for wired charging. Alternatively, the writing pen can be placed in the accompanying charging case for charging via contact points.

[0127] In some embodiments, the writing pen body also includes an indicator light and a power switch. For example, the indicator light is located in a visible position on the writing pen body and displays the current operating status (e.g., powered on, recording, low battery, data transmission, charging, etc.) through different colors or flashing patterns. The power switch is located on the writing pen body and is turned on or off by the user pressing and holding or flicking it. In some embodiments, the writing pen can enter standby mode upon receiving a voice wake-up word.

[0128] The following details the working process of the writing pen: Upon power-on, the indicator light illuminates, and the writing pen enters standby mode. The writing signal detection module outputs a pressure signal to the main control chip based on the detected pressure data between the pen tip and the writing surface. Upon receiving the pressure signal, the main control chip controls the image acquisition module to capture images of the writing area near the pen tip frame by frame at a preset acquisition rate (e.g., 3 frames per second), and controls the audio acquisition module to acquire surrounding audio data, adding a timestamp to each frame of the writing image and each segment of audio data. The main control chip can then send the audio data and corresponding timestamps to the control terminal and / or server. When the writing signal detection module detects that the pressure data between the pen tip and the writing surface is 0 (i.e., the pen is lifted), and after maintaining this position for a preset time threshold, the main control chip can control the image acquisition module and audio acquisition device to automatically enter a low-power sleep mode.

[0129] The main control chip performs at least one of the following preprocessing operations on the acquired raw handwriting image: image stabilization, image enhancement, distortion correction, and data compression. Image stabilization can be understood as detecting and compensating for image shifts caused by hand tremors using an image registration algorithm; image enhancement may include at least one of noise reduction, contrast enhancement, and brightness correction; distortion correction can be understood as correcting the distortion of the raw handwriting image based on a distortion model of a wide-angle lens; and data compression can be understood as compressing and encoding the preprocessed image. The preprocessed handwriting image and its corresponding timestamp can be stored in a storage module and sent to the control terminal.

[0130] The raw audio data contains various noise sources (pen tip friction noise, paper turning noise, ambient noise, etc.), therefore, multi-level audio preprocessing is required to improve the audio acquisition quality. In some embodiments, various main control chips perform at least one of the following preprocessing operations on the acquired raw audio data: noise reduction, pen tip friction noise suppression, echo cancellation, and automatic gain control. Specifically, noise reduction can involve performing a short-time Fourier transform on the audio signal, estimating the noise power spectrum in the frequency domain, removing steady-state noise through spectral subtraction and a Wiener filter, and achieving a balance between noise reduction and speech distortion. Pen tip friction noise suppression can adaptively increase the noise reduction intensity when the pressure data detected by the pressure sensor increases; decrease the noise reduction when the pen tip is lifted; and dynamically adjust the threshold for speech activity detection based on the current writing state. Echo cancellation can estimate the acoustic transfer function from the speaker to the microphone using an adaptive filter, generate an echo estimation signal, and subtract the echo estimation signal from the acquired signal. Automatic gain control can automatically adjust the audio gain through an Automatic Gain Control (AGC) module.

[0131] In some embodiments, a dual-channel transmission mechanism is used between the writing pen and the control terminal, where audio data and writing images are transmitted as independent data streams. The transmission priority of the writing signal is higher than that of the writing image, which in turn is higher than the audio data, and the audio data is higher than other data. In cases of insufficient bandwidth, the resolution and frame rate of the writing image can be reduced first, followed by a reduction in the audio encoding bit rate. Finally, only the pixel coordinates of key trajectory points and low-quality audio data can be transmitted, with the writing image stored locally and then synchronized in batches. When the wireless connection is interrupted, both audio data and writing images are locally cached in the writing pen's storage module and automatically synchronized upon reconnection, ensuring no data loss. Data packets use a unified encapsulation format, including a packet header, timestamp, type flag (writing trajectory / audio frame / synchronization flag / image frame), data length, and payload.

[0132] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0133] Based on the same inventive concept, this application also provides a content generation apparatus for implementing the content generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more content generation apparatus embodiments provided below can be found in the limitations of the content generation method described above, and will not be repeated here.

[0134] In one exemplary embodiment, such as Figure 10 As shown, a content generation device is provided, configured in a control terminal, including: a first receiving module 1010, a first determining module 1020, a second determining module 1030, a third determining module 1040, and a generation module 1050, wherein:

[0135] The first receiving module 1010 is used to receive at least one frame of writing image captured by the writing pen during the writing process;

[0136] The first determining module 1020 is used to determine the global writing image of the writing pen during the writing process based on at least one frame of writing image;

[0137] The second determining module 1030 is used to determine the writing trajectory of the writing pen during the writing process, as well as the pixel coordinate data of each trajectory point on the writing trajectory, based on the global writing image.

[0138] The third determining module 1040 is used to determine the connection relationship between the trajectory points based on the writing order of each trajectory point; and

[0139] The generation module 1050 is used to generate target text content based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point.

[0140] In one embodiment, the first determining module 1020 is specifically configured to: perform feature point matching on any two adjacent writing images to obtain feature point pairs in the two writing images; determine the coordinate transformation relationship between the two writing images based on the feature point pairs; perform coordinate transformation on the two writing images based on the coordinate transformation relationship to obtain the target writing images corresponding to each of the two writing images in the same coordinate space; and perform fusion processing on the obtained target writing images to obtain the global writing image of the writing pen during the writing process.

[0141] In one embodiment, the second determining module 1030 is specifically used for: performing image recognition on the global writing image to obtain the writing trajectory of the writing pen during the writing process; performing binarization processing on the global writing image to obtain a binarized writing image; and extracting pixel coordinate data of each trajectory point on the writing trajectory from the binarized writing image.

[0142] In one embodiment, the generation module 1050 is specifically used to: acquire audio text content; wherein the audio text content is obtained by converting audio data collected by the writing pen during the writing process into text; and generate target text content based on the audio text content, the pixel coordinate data of each trajectory point, and the connection relationship between each trajectory pixel point.

[0143] In one embodiment, the generation module 1050 is specifically used to: receive audio data collected by the writing pen during the writing process, convert the audio data into text, and obtain audio text content.

[0144] In one embodiment, the generation module 1050 is specifically used to: obtain the target document type; and generate target text content that matches the target text type based on the audio text content, the pixel coordinate data of each trajectory point, and the connection relationship between each trajectory pixel point.

[0145] In one exemplary embodiment, such as Figure 11 As shown, a content generation device is provided, configured on a stylus, including: a first acquisition module 1110 and a first transmission module 1120, wherein:

[0146] The first acquisition module 1110 is used to acquire at least one frame of writing image captured by the writing pen during the writing process in response to the writing signal of the writing pen.

[0147] The first sending module 1120 is used to send at least one frame of the written image to the control terminal;

[0148] Among them, at least one frame of writing image is used for the control terminal to determine the global writing image of the writing pen in the writing process; based on the global writing image, the writing trajectory of the writing pen in the writing process and the pixel coordinate data of each trajectory point on the writing trajectory are determined; based on the writing order of each trajectory point, the connection relationship between each trajectory point is determined; and based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point, the target text content is generated.

[0149] In one embodiment, the content generation device further includes: a second acquisition module for acquiring audio data collected by the writing pen during the writing process; and a second sending module for sending the audio data to a control terminal.

[0150] Each module in the aforementioned content generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0151] In one exemplary embodiment, a writing pen is provided, the internal structure of which can be shown in the diagram below. Figure 12 As shown, the writing pen includes a processor and memory. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium storing a computer program. When executed by the processor, the computer program implements a writing method.

[0152] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the writing pen on which the present application is applied. A specific writing pen may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0153] In one exemplary embodiment, a writing pen is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0154] In response to the writing signal from the writing pen, acquire at least one frame of writing image captured by the writing pen during the writing process;

[0155] Send at least one frame of the written image to the control terminal;

[0156] Among them, at least one frame of writing image is used for the control terminal to determine the global writing image of the writing pen in the writing process; based on the global writing image, the writing trajectory of the writing pen in the writing process and the pixel coordinate data of each trajectory point on the writing trajectory are determined; based on the writing order of each trajectory point, the connection relationship between each trajectory point is determined; and based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point, the target text content is generated.

[0157] In one embodiment, the processor executes the following steps when running a computer program: acquiring audio data collected by the writing pen during the writing process; and sending the audio data to a control terminal.

[0158] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the content generation method provided in any of the above embodiments.

[0159] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the content generation method provided in any of the above embodiments.

[0160] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0162] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A content generation method, characterized in that, Applied to a control terminal, the method includes: Receive at least one frame of writing image captured by the writing pen during the writing process; Based on the at least one frame of writing image, determine the global writing image of the writing pen during the writing process; Based on the global writing image, determine the writing trajectory of the writing pen during the writing process, as well as the pixel coordinate data of each trajectory point on the writing trajectory; Based on the writing order of the trajectory points, the connection relationships between the trajectory points are determined; and The target text content is generated based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between the trajectory points.

2. The method according to claim 1, characterized in that, Determining the global writing image of the writing pen during the writing process based on the at least one frame of writing image includes: For any two adjacent frames of writing images, feature point matching is performed on the two frames of writing images to obtain feature point pairs in the two frames of writing images; Based on the feature point pairs, determine the coordinate transformation relationship between the two frames of written images; Based on the coordinate transformation relationship, the two frames of writing images are transformed to obtain the target writing images corresponding to each of the two frames in the same coordinate space; and The obtained target writing image is fused to obtain the global writing image of the writing pen during the writing process.

3. The method according to claim 1, characterized in that, The step of determining the writing trajectory of the writing pen during the writing process, and the pixel coordinate data of each trajectory point on the writing trajectory, based on the global writing image, includes: Image recognition is performed on the global writing image to obtain the writing trajectory of the writing pen during the writing process; The global writing image is binarized to obtain a binary writing image; and Pixel coordinate data of each trajectory point on the writing trajectory are extracted from the binarized writing image.

4. The method according to any one of claims 1 to 3, characterized in that, The step of generating target text content based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between the trajectory points includes: Acquire audio text content; wherein, the audio text content is obtained by converting the audio data collected by the writing pen during the writing process into text; The target text content is generated based on the audio text content, the pixel coordinate data of each trajectory point, and the connection relationship between each trajectory pixel point.

5. The method according to claim 4, characterized in that, The acquisition of audio text content includes: Receive audio data collected by the writing pen during the writing process; The audio data is converted into text to obtain the audio text content.

6. The method according to claim 4, characterized in that, The step of generating target text content based on the audio text content, the pixel coordinate data of each trajectory point, and the connection relationship between each trajectory pixel point includes: Obtain the target document type; Based on the audio text content, the pixel coordinate data of each trajectory point, and the connection relationship between each trajectory pixel point, target text content matching the target text type is generated.

7. A content generation method, characterized in that, Applied to a writing pen, the method includes: In response to the writing signal of the writing pen, at least one frame of writing image captured by the writing pen during the writing process is acquired; Send the at least one frame of the written image to the control terminal; Wherein, the at least one frame of writing image is used by the control terminal to determine the global writing image of the writing pen in the writing process; based on the global writing image, the writing trajectory of the writing pen in the writing process, and the pixel coordinate data of each trajectory point on the writing trajectory are determined; based on the writing order of each trajectory point, the connection relationship between each trajectory point is determined; and based on the pixel coordinate data of each trajectory point on the writing trajectory and the connection relationship between each trajectory point, the target text content is generated.

8. The method according to claim 7, characterized in that, Also includes: Acquire audio data collected by the writing pen during the writing process; The audio data is sent to the control terminal.

9. A writing pen, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 7 or 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6 or the steps of the method according to claim 7 or 8.

Images