Target character stroke-by-stroke display method, system and device based on voice semantic analysis

Abstract

The application provides a kind of based on speech semantic analysis letter by letter display method, system and equipment, comprising the following steps: S1: receiving user voice instruction, speech activity detection is carried out to voice instruction, when detecting continuous silence reaches preset time length, end speech acquisition, the preset time length is 600~1200ms;S2: the semantic analysis of the collected speech text, based on the rule template, the target text is parsed from the speech text;S3: the voice segment, recognized text or target text is provided to writing guide data generation module, and the writing guide data corresponding to the target text is obtained;S4: on display device, based on guide layer data display background guide word and auxiliary grid line, and based on the stroke data stroke by stroke superimposed display current stroke.The application can reduce the threshold of use, provide coherent writing guide experience, and adapt to the display characteristics of display device.

Description

Technical Field

[0001] This invention relates to the field of text writing assistance technology, and in particular to a method, system and device for displaying text stroke by stroke based on speech semantic analysis. Background Technology

[0002] With the development of smart educational hardware, the demand for handwriting practice devices for children is increasing. Existing technical solutions typically suffer from the following problems: 1. High input threshold: It is inconvenient for users such as children to manually input complex Chinese characters via keyboard or touch screen.

[0003] 2. Fragmented technology chain: Existing voice interaction functions and professional calligraphy template generation and stroke order guidance functions are usually located in different applications or devices, failing to form a smooth end-to-end experience.

[0004] 3. Lack of in-depth optimization for low-power hardware: Many calligraphy practice applications or animation designs are geared towards high-performance color screens and do not fully consider the unique display characteristics of e-ink screens, such as low refresh rates, power sensitivity, and limited grayscale performance (especially in low bit depth mode with limited grayscale levels). Directly applying general image display solutions will result in slow refresh rates, severe ghosting, and inability to achieve effective grayscale guidance on e-ink screens, affecting the writing practice experience. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, and device for displaying text stroke by stroke based on speech semantic parsing, so as to reduce the threshold of use, provide a coherent writing guidance experience, and adapt to the display characteristics of display devices.

[0006] The core process of this invention is as follows: extract target characters from natural language, automatically generate stroke-by-stroke writing guidance data adapted to the display device, and then display the guidance stroke by stroke.

[0007] To achieve the above objectives, the solution of the present invention is as follows: In a first aspect, the present invention provides a method for displaying text stroke by stroke based on speech semantic parsing, comprising the following steps: S1: Receive user voice commands, perform voice activity detection on the voice commands, and end voice acquisition when continuous silence is detected for a preset duration; Among them, voice activity detection is performed on the device and / or cloud server, with a preset duration of 600~1200ms; S2: Perform semantic parsing on the collected speech text, and extract the target text from the speech text based on the preset rule template; S3: Provide the speech segment, recognized text, or target text to the writing guidance data generation module, and obtain the writing guidance data corresponding to the target text; The writing guidance data includes guidance layer data and stroke-by-stroke data, and the writing guidance data is one or more of the following: bitmap data, SVG path data, vector stroke data, center line trajectory data, and compressed frame data. The writing guidance data is generated through one or more of the following methods: character library retrieval, font parsing, vector path conversion, or artificial intelligence model generation. S4: On the display device, the background guide text and auxiliary grid lines are displayed based on the guide layer data, and the current stroke is displayed by overlaying the stroke data stroke by stroke.

[0008] In a preferred embodiment, the target text includes Chinese characters, letters, numbers, or symbols.

[0009] In a preferred embodiment, in step S1, the speech activity detection employs an energy threshold method, the parameters of which include: The frame length is 20~30ms, the threshold for starting to speak is -45dBFS~-35dBFS, and the threshold for ending silence is -50dBFS~-40dBFS.

[0010] In a preferred embodiment, in step S2, the rule template includes a sentence structure for extracting target text from natural language.

[0011] In a preferred embodiment, the triggering method for displaying each item in step S4 includes at least one of the following: receiving physical button signals, receiving voice commands, or automatically switching at preset time intervals.

[0012] In a preferred embodiment, the bootstrap data is generated locally by a cloud server or the device.

[0013] In a preferred embodiment, the stroke-by-stroke data is a low-bit-depth bitmap format adapted to the display device; the display device includes an electronic ink screen, a thermal printer, or electronic paper.

[0014] In a preferred embodiment, step S4 further includes displaying the phonetic annotation of the target text; The stroke-by-stroke data is a 1-bit-per-pixel bitmap frame, and the guide layer data is data generated by dot matrix dithering to simulate grayscale on a 1-bit-per-pixel display device.

[0015] Secondly, the present invention provides a text-by-text display system based on speech semantic parsing, comprising: The voice acquisition and endpoint detection module is used to receive user voice commands and detect voice activity. The acquisition ends when continuous silence is detected for 600~1200ms. The voice acquisition and endpoint detection module supports local endpoint detection or streaming to the cloud for endpoint detection. The parsing and data generation module is used to recognize the collected speech, parse the target text based on the preset rule template, and generate the stroke data and guidance layer data of the target text. The device-side display control module is used to receive stroke-by-stroke drawing data and guide layer data, overlay them on the display device, and trigger the display stroke by stroke.

[0016] Thirdly, the present invention provides a target text writing practice device, including a main controller, a memory, a microphone, a display device, a speaker module, and a wireless communication module; The main controller is communicatively connected to the memory, microphone, display device, speaker module, and wireless communication module, respectively. The memory stores a computer program, which, when executed by the main controller, implements the above-mentioned method for displaying text stroke by stroke based on speech semantic parsing.

[0017] In a preferred embodiment, the main controller is an ESP32 series microcontroller, the display device is an electronic ink screen, and the driver chip of the electronic ink screen is an ST8899.

[0018] After adopting the above solution, the beneficial effects of the present invention are as follows: 1. Natural and easy-to-use interactive experience: Users can specify the text they want to practice through natural voice, without the need for manual input, which greatly reduces the barrier to entry and is especially suitable for children.

[0019] 2. Highly Efficient Edge-Cloud Collaborative Processing Architecture: This invention supports the flexible deployment of functions such as speech recognition, text parsing, and image generation on servers or devices for local execution, with the device responsible for data acquisition, communication, and display. This architecture allows terminal devices to select the most suitable computing resource configuration based on actual computing power and network conditions, achieving an optimized balance between performance and cost.

[0020] 3. Deeply Adapted Display Optimization for E-Ink Screens: This invention provides speech fragments, recognized text, or target text to the writing guidance data generation module, obtains writing guidance data corresponding to the target text, and overlays it on the display device. This solves the problem of mismatch between general image data and display device hardware characteristics from the data source. This method generates small amounts of display data with high refresh efficiency, effectively eliminates display ghosting, and achieves clear stroke order guidance on the display device, significantly improving the writing guidance experience on low-power display devices.

[0021] 4. Guided teaching that conforms to writing habits: This invention displays background guide characters and auxiliary grid lines based on the guidance layer data, and displays the current stroke stroke by stroke, providing users with a "tracing" reference and dynamic stroke order demonstration, forming a complete and highly interactive writing practice loop, with intuitive and effective teaching results. Attached Figure Description

[0022] Figure 1 This is a system interaction flowchart in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the effect of stroke-by-stroke display and the overlay of the guide layer in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0024] This embodiment provides a method for displaying text stroke by stroke based on speech semantic parsing, including the following steps: S1: Receive user voice commands, perform voice activity detection on the voice commands, and end voice acquisition when continuous silence is detected for a preset duration; Among them, voice activity detection is performed on the device and / or cloud server, with a preset duration of 600~1200ms; S2: Perform semantic parsing on the collected speech text, and extract the target text from the speech text based on the preset rule template; S3: Provide the speech segment, recognized text, or target text to the writing guidance data generation module, and obtain the writing guidance data corresponding to the target text; The writing guidance data includes guidance layer data and stroke-by-stroke data, and the writing guidance data is one or more of the following: bitmap data, SVG path data, vector stroke data, center line trajectory data, and compressed frame data. The writing guidance data is generated through one or more of the following methods: character library retrieval, font parsing, vector path conversion, or artificial intelligence model generation. S4: On the display device, the background guide text and auxiliary grid lines are displayed based on the guide layer data, and the current stroke is displayed by overlaying the stroke data stroke by stroke.

[0025] The writing guidance data generation module described in this embodiment is a functional module used to generate writing guidance data based on the target text. The writing guidance data includes guidance layer data and stroke-by-stroke data, which are used to present the calligraphy template and stroke guidance information on the display device.

[0026] The writing guidance data generation module can be deployed on a cloud server, a local server, a gateway device, or locally on the device itself. The specific deployment method depends on the system's computing power configuration and network conditions. When deployed on a cloud server, the device sends voice clips, recognized text, or target text to the cloud via the network, and the cloud generates writing guidance data and sends it back to the device. When deployed locally on the device, the device completes the parsing of the target text and the generation of writing guidance data locally, without relying on a network connection; When a hybrid deployment is adopted, some functions (such as speech recognition) are executed in the cloud, while some functions (such as writing guidance data generation) are executed locally.

[0027] Regardless of the deployment method used, the core function is to generate corresponding writing guidance data based on the target text. This embodiment uses an e-ink screen as an example for description.

[0028] Step S1: Voice Acquisition and Endpoint Detection The device's microphone continuously collects ambient audio. Upon detecting a preset wake-up word, it enters voice command reception mode and then initiates voice activity detection. This voice activity detection can be performed on the device itself, on a cloud server, or jointly by the device and the cloud server.

[0029] In this embodiment, Voice Activity Detection (VAD) is performed on the device side, using the energy threshold method for endpoint detection.

[0030] Upon wake-up, the device enters a low-power listening state. In this embodiment, an energy threshold method combined with an audio front-end algorithm (such as the VAD module in the ESP-SR / AFE solution) is preferably used for voice endpoint detection. The specific judgment logic includes: 1. Parameter settings: To ensure the sensitivity and stability of detection, the frame length is set to 20~30ms; the threshold for starting to speak is set to a short-term energy higher than -45dBFS~-35dBFS; the threshold for ending silence is set to a short-term energy lower than -50dBFS~-40dBFS; and the duration of ending silence is set to 600~1200ms, preferably 800ms in this embodiment.

[0031] 2. Judgment process: When the device detects that the audio energy value of several consecutive frames (e.g., 3 frames) exceeds the start speaking threshold, it determines that the user has started speaking and then starts recording; during the recording process, if the audio energy value of several consecutive frames (e.g., 5 frames) is detected to be lower than the end silence threshold, and the duration of this low energy state exceeds the preset end silence duration (600~1200ms), it determines that the voice command has ended, then the recording ends and the valid voice segment is uploaded to the cloud server.

[0032] With the above configuration, the user's complete instructions can be accurately captured, and environmental noise and invalid data during user pauses can be effectively filtered out, reducing network transmission pressure and improving response efficiency.

[0033] In other embodiments, voice activity detection is performed on a cloud server. After the device is woken up, it uploads the collected PCM audio frame data stream to the cloud server in real time; the cloud server performs real-time voice activity detection on the received audio stream to determine the start and end of speech; when the cloud VAD detects continuous silence for a preset duration (600~1200ms), it determines that the speech ends; the cloud server performs speech recognition and target text parsing on the received complete audio. This embodiment is applicable to scenarios where the computing power of the device is limited or it is desired to simplify the logic of the device.

[0034] Step S2: parsing the target text.

[0035] Taking a cloud server as an example, after the cloud server receives audio data, it first calls an automatic speech recognition service to convert the audio data into text. Then, a parsing service performs semantic parsing on the text based on a preset rule template.

[0036] For example, the rule template may be defined as: matching the sentence pattern of "I want to write [word]'s [target Chinese character]", and extracting the content at the specified position as the target Chinese character; or matching the sentence pattern of "I want to write [word]", and extracting the content at the specified position as the target Chinese character. In this way, for example, the target character "祖" can be accurately parsed from the speech recognition text "I want to write the 'zu' of the motherland". The target text is not limited to Chinese characters, and may also be English letters, numbers or other symbols.

[0037] Taking Chinese characters as an example, the target Chinese character parsing method includes the following steps: ① Normalizing the speech recognition result text, removing punctuation, spaces and colloquialisms to obtain a normalized text; ② Matching preset rule templates in sequence; ③ When matching the "I want to write Y of X" type template, extracting X and Y; ④ When Y is a single Chinese character, determining Y as the target Chinese character; ⑤ When Y is a position description, extracting the corresponding Chinese character from X as the target Chinese character according to the position description; ⑥ When the "I want to write Y of X" type template is not matched, matching the "I want to write X" type template; ⑦ When X is a single Chinese character, determining X as the target Chinese character; ⑧ When X is a multi-character word, taking the first character as the target Chinese character, or generating a candidate target Chinese character list; ⑨ When the rule template cannot uniquely determine the target Chinese character, calling a large model for auxiliary discrimination; ⑩ Performing constraint verification on the output result of the large model, if the preset condition is met, outputting the target Chinese character, otherwise returning a clarification prompt or re-recognizing.

[0038] Exemplary pseudocode is as follows: function parse_target_char(asr_text): text = normalize(asr_text) # Rule Template 1: I want to write the Y of X if regex_match(text, "I want to write (.*) of (.)"): X = regex_group(text, 1) Y = regex_group(text, 2) if is_single_hanzi(Y): return Y if is_position_word(Y): return pick_char_by_position(X, Y) # Rule Template 2: I want to write X if regex_match(text, "I want to write (.+)"): X = regex_group(text, 1) if is_single_hanzi(X): return X if is_hanzi_phrase(X): return first_char(X) candidate_list = build_candidate_list(X) if length(candidate_list) == 1: return candidate_list[0] if length(candidate_list)>1: return candidate_list # When the rule template is not matched, call the large model to assist in the judgment. llm_result = llm_parse_target_char(text) if is_valid_target_char(llm_result): return llm_result return need_clarification function llm_parse_target_char(text): prompt = """ You are the target Chinese character parsing module.

[0039] Only one candidate target character is allowed to be output.

[0040] The input text may contain phrases such as "I want to write Y of X" or "I want to write X".

[0041] It can only output a single Chinese character or a predefined writable character.

[0042] """ result = call_llm(prompt, text) return result function is_valid_target_char(ch): if not is_single_hanzi_or_allowed_char(ch): return false if not exists_in_stroke_dictionary(ch): return false if confidence(ch) <THRESHOLD: return false return true function pick_char_by_position(X, Y): if Y == "first character" or Y == "starting character": return char_at(X, 0) if Y == "second character": return char_at(X, 1) if Y == "last word" or Y == "last character": return char_at(X, length(X) - 1) if Y == "the middle character": return char_at(X, floor(length(X) / 2)) return null Wherein, the implementation of the position description parsing function pick_char_by_position is as follows: If Y is "first character" or "starting character", return the first character of X; If Y is "second character", return the second character of X; If Y is "last character" or "ending character", return the last character of X; If Y is "middle character", return the character at the middle position of X.

[0043] The large model auxiliary discrimination function llm_parse_target_char calls a large language model through a preset prompt, requiring it to output only one candidate target character. The output result also needs to be verified by the constraint check function is_valid_target_char, including: whether it is a single Chinese character or an allowed character, whether it exists in the stroke order dictionary, whether the confidence level meets the threshold, etc.

[0044] Step S3: Generate writing guidance data Specifically, after parsing the target text (e.g., "祖"), the writing guidance data generation module performs the following operations: The writing guidance data is generated in one or more of the following ways: ① Font library retrieval: According to the target text, retrieve the corresponding glyph data in a preset font library file. The font library file may be a bitmap font library, a vector font library (such as TrueType, OpenType) or a custom-format font library. After the glyph data is retrieved, the stroke order information and stroke paths are further extracted.

[0045] ② Font parsing: When the font file corresponding to the target text is available, parse the font file through a font parsing engine (such as FreeType, Skia) to extract the outline path and metric information of the character, and then generate stroke-by-stroke data and guide layer data.

[0046] ③ Vector path conversion: Perform coordinate transformation, scaling and rasterization processing on the obtained vector path data (such as SVG format), and convert it into bitmap data adapted to a display device or writing guidance data in other formats.

[0047] ④ Artificial intelligence model generation: corresponding writing guidance data is directly generated according to the target text through a trained deep learning model (such as GAN, Transformer, etc.). Said model can be trained based on a large amount of copybook data, and can generate writing guidance data with uniform style and accurate stroke order.

[0048] The above generation methods can be used alone or in combination. For example, a basic glyph can be obtained first through font library retrieval, then stroke order information is supplemented through font parsing, and finally the final writing guidance data is generated through vector path conversion.

[0049] 1. Query basic information: According to the target text, query its corresponding vector glyph contour file (such as SVG format) and standard stroke order information; for other characters such as English, query its standard writing trajectory data.

[0050] 2. Generate stroke-by-stroke data: Said stroke-by-stroke data can be one or more of bitmap data, SVG path data, vector stroke data, centerline trajectory data, and compressed frame data. This embodiment takes bitmap data as an example, and the specific steps are as follows: ① Obtain vector glyph data and writing order information of the target character; ② Split the target character into a plurality of sequentially arranged stroke paths according to the writing order information; ③ Perform coordinate transformation, scaling and centering processing on each stroke path to adapt it to a preset display canvas; ④ Rasterize and render the processed stroke paths into a bitmap with a specified size; ⑤ Organize a plurality of single-stroke bitmaps into a bitmap frame sequence according to the writing order; ⑥ Attach frame header metadata to each bitmap frame, including stroke sequence number, bitmap width, bitmap height, data length, etc.; ⑦ Package the frame header and bitmap data and send them to the display module through wireless communication; ⑧ The display module decodes and displays the bitmap frames sequentially according to the frame sequence number, so as to realize stroke-by-stroke writing guidance.

[0051] Stroke splitting of Chinese characters is implemented based on a preset stroke order data table, vector contour library or Chinese character writing data set. Specifically, the system pre-stores a stroke list for each target Chinese character, each stroke corresponds to an independent vector path, contour path or centerline trajectory, and is sorted according to the preset writing order. When generating bitmap frames, the system sequentially takes out the path set of the n-th stroke or the first n strokes according to the stroke order to form the display content of the current step.

[0052] For example, for the Chinese character "祖", the system pre-stores its stroke sequence as: 1st stroke → Path 1; Second stroke → Path 2; ...; Nth stroke → path N.

[0053] Each path can be represented as an SVG path, a Bezier curve, a set of line segments, a set of polylines, or a set of centerline sampling points.

[0054] For other languages ​​such as English, the system can use one of the following methods to split the strokes: ①If the target character comes from a font outline library, the letter outline can be split into stroke paths arranged in the writing order according to the preset letter writing trajectory table; ②If the target character comes from handwritten trajectory data, the handwritten sampling trajectory can be used directly as the stroke path; ③ If there is only the outline of the character but no stroke order information, the outline of the character can be converted into stroke trajectories arranged in sequence by means of skeleton extraction, centerline decomposition, connected component decomposition or preset letter stroke order rules. ④ For English letters, the system can create a stroke order table based on letter writing habits. For example, the sequential strokes corresponding to characters such as A, B, and C, or the characters can be broken down according to the order of the first stroke, main vertical stroke, horizontal stroke, and last stroke.

[0055] During rendering, the system first scales, translates, and centers the vector path of the target character to ensure it falls within a preset display area (e.g., 400 pixels × 300 pixels); then, it rasterizes the path, overlaying it onto a pixel grid. This can be achieved using FreeType, Skia, Cairo, Pango, Pillow, OpenCV, AGG, or other graphics libraries that support vector drawing and rasterization.

[0056] Rendering methods may include the following steps: ① Calculate the original bounding box of the target character path; ② Calculate the scaling ratio based on the preset canvas size; ③ Move the path to the center of the canvas or the center of the grid lines; ④ Draw the path using either anti-aliasing or non-anti-aliasing; ⑤ If the target is a 1bpp bitmap, then the grayscale result will be further binarized; ⑥ Set pixels with values ​​greater than the threshold to 1 and pixels with values ​​less than the threshold to 0 to form a pure black and white bitmap.

[0057] 3. Generate bootstrap layer data: The guiding layer is used to provide background guidance on the display device, preferably gray, but not limited to gray. This embodiment uses an ordered dithering algorithm to convert the grayscale bitmap into a 1-bit-per-pixel black-and-white bitmap. This ordered dithering can be implemented using a Bayer matrix or a variant thereof. Ordered dithering is chosen because it has low computational cost, is simple to implement, provides stable output, and offers high visual consistency between frames. In other embodiments, an error diffusion dithering algorithm, such as the Floyd-Steinberg algorithm, can also be used.

[0058] The input to the dithering algorithm is a complete grayscale glyph bitmap, which can be an 8-bit grayscale image, and is preferably a glyph image after anti-aliasing rendering. The steps to convert the grayscale image into a 1-bit-per-pixel black and white bitmap include: ① Obtain an 8-bit grayscale image with pixel values ​​ranging from 0 to 255; ② Read the current pixel coordinates (x, y); ③ Extract the corresponding threshold from the ordered jitter matrix based on the pixel coordinates; ④ Compare the current pixel grayscale value (gray) with the threshold. ⑤ If gray is less than threshold, output 0 black pixels; otherwise, output 1 white pixel. ⑥ Pack the 8 output pixels bit by bit into 1 byte to form 1bpp bitmap data; ⑦ Repeat the above steps until the entire grayscale image is converted.

[0059] The following is an example pseudocode: { function dither_to_1bpp(gray_image): width = gray_image.width height = gray_image.height output = new byte[(width * height) / 8] threshold_matrix = [ [64, 192], [224, 128] ] for y in range(0, height): for x in range(0, width): gray = gray_image.pixel(x, y)# 0..255 threshold = threshold_matrix[y % 2][x % 2] black = (gray <threshold) byte_index = (y * width + x) / 8 bit_index = 7 - (x % 8) if not black: output[byte_index] |= (1< <bit_index) return output } In other embodiments, if a standard Bayer matrix is ​​used, a 4x4 or 8x8 threshold matrix can be used instead of the 2x2 matrix described above, but the basic processing logic is the same, and they all fall under the category of ordered jitter.

[0060] The organization of the guide layer bitmap and the step-by-step bitmap frames can be done in one of the following two ways: Independent frame mode: The guide layer, acting as a separate base frame, is generated by the server or locally and sent and displayed first. Subsequently, the stroke-by-stroke frame sequence is sent sequentially as differential frames. When displaying the image, the device overlays the currently received stroke frames with the already displayed guide layer to form the final image.

[0061] Packaging method: The guide layer and stroke frame sequence are pre-calculated and synthesized into a complete frame sequence on the server or locally. That is, each frame is already the final image containing the guide layer and the current stroke layer. The device does not need to perform any overlay calculations and only needs to display them in sequence according to the frame number.

[0062] This embodiment preferably uses a packetization method, where the guide layer is used as the background layer of the current display frame, and it is combined with the current stroke layer to form a complete bitmap data frame before being sent together. In another embodiment, vector path data (such as SVG format) can also be used instead of bitmap frames for transmission and display.

[0063] 4. Data encapsulation and distribution: The generated writing guidance data is packaged according to a predefined communication protocol and sent to the display module. The writing guidance data may include the target text, total number of strokes, guidance layer data, stroke-by-stroke data, canvas parameters, rendering parameters, and verification information.

[0064] When the writing guidance data generation module and the display module are located in the same device, the two may transmit data via memory cache, files, message queues, inter-process communication or an internal device bus; when the writing guidance data generation module is deployed on the server side, data may be transmitted via WebSocket, TCP, HTTP long connection or other two-way communication methods. The transmission protocol may adopt JSON, TLV or binary frame format.

[0065] ① Example of JSON format: { "type": "copybook_payload", "version": 1, "request_id": "req_20260610_0001", "target_char": "祖", "pinyin": "zu3", "language": "zh-CN", "view_mode": "practice", "canvas": { "width": 400, "height": 300 , "stroke_count": 9, "guide_layer": { "layer_type": "gray_guide", "format": "1bpp", "encoding": "base64", "width": 400, "height": 300, "data_length": 15000, "bitmap_1bpp_base64": "..." , "stroke_frames": { "frame_index": 0, "stroke_index": 1, "stroke_count": 9, "width": 400, "height": 300, "format": "1bpp", "encoding": "base64", "data_length": 15000, "bitmap_1bpp_base64": "..." }, { "frame_index": 1, "stroke_index": 2, "stroke_count": 9, "width": 400, "height": 300, "format": "1bpp", "encoding": "base64", "data_length": 15000, "bitmap_1bpp_base64": "..." }, { "frame_index": 2, "stroke_index": 3, "stroke_count": 9, "width": 400, "height": 300, "format": "1bpp", "encoding": "base64", "data_length": 15000, "bitmap_1bpp_base64": "..." } ], "status": { "wifi_rssi": -55, "battery_percent": 82, "backend_connected": true, "vad_state": "idle" } } The meanings of each field are as follows: target_char: Target text; pinyin: Phonetic annotation of target text, such as "zu3", used to display pronunciation information on display devices; stroke_count: Total number of strokes; guide_layer: Gray guide layer data; stroke_frames: An array of stroke frames arranged in order; canvas.width and canvas.height: Size of the bitmap canvas; format: pixel format, preferably 1bpp; encoding: encoding method, preferably base64; data_length: Bitmap data length, used for device-side verification; frame_index and stroke_index: used to ensure display order and restore playback status.

[0066] ② TLV format example In other embodiments, the protocol can be described using a TLV structure, i.e., Type-Length-Value. Specifically: message header - message_type: 1 byte - version: 1 byte - request_id_length: 2 bytes - request_id: Variable length - payload_length: 4 bytes Load field - type = 0x01: target_char - length: 2 - value: UTF-8 encoded target text - type = 0x02: stroke_count - length: 2 - value: Total number of strokes - type = 0x03: guide_layer - length: variable - value: Gray guide layer bitmap data - type = 0x04: stroke_frame_count - length: 2 - value: Number of stroke frames - type = 0x05: stroke_frame_array - length: variable - value: Sequential splicing of multiple stroke frames - type = 0x06: canvas_width - length: 2 - value: 400 - type = 0x07: canvas_height - length: 2 - value: 300 - type = 0x08: bitmap_encoding - length: 1 - value: 0 indicates raw 1bpp, 1 indicates base64, 2 indicates RLE - type = 0x09: frame_index - length: 2 - value: Current frame number - type = 0x0A: checksum - length: 4 - value: CRC32 or other checksum The stroke frame array can be organized using either a cumulative or incremental method. In the cumulative method, the nth frame contains the composite result of the previous 1 to n strokes, meaning each frame is a complete image. In the incremental method, the nth frame only contains the changed area of ​​the nth stroke, which is superimposed onto the previous frame by the device. This embodiment preferably uses a cumulative structure. Step S4: Device-side display and interaction After receiving the data packet, the device parses and stores it. Then: 1. Initialize display: Refresh the guide layer data to the display device in full screen as a fixed background guide text and auxiliary grid lines.

[0067] 2. Displaying Phonetic Markings: The phonetic markings of the target text are also displayed on the display device. These phonetic markings include, but are not limited to, Pinyin, International Phonetic Alphabet, Zhuyin Fuhao, or pronunciation markings in other languages. The preferred display position of the phonetic markings is above or below the target text, but they can also be displayed in a designated area of ​​the display device (such as the top or bottom). The font size, color, and style of the phonetic markings can be adjusted according to the characteristics of the display device and user preferences. The phonetic marking data can be generated and distributed by the server when generating writing guidance data (such as the "pinyin" field in the aforementioned JSON example), or it can be obtained by the device through local querying based on the target text.

[0068] 3. Waiting for trigger: The device enters interactive mode and waits for the user to trigger the display of each item.

[0069] 4. When the display is triggered, the device's main controller overlays the current stroke's bitmap frame with the background guide layer to generate new screen buffer data and drives the display device to refresh. In this embodiment, a display device supporting partial refresh (such as an e-ink screen) is used. Figure 2 As shown, only the display area corresponding to the current stroke is partially refreshed, instead of the entire screen. Specifically, the device's main controller calculates the bounding box of the stroke on the screen based on the bitmap frame of the current stroke, and then only refreshes the pixels within that bounding box to reduce refresh power consumption and shorten refresh time. For display devices that do not support partial refresh, a global refresh method is used, that is, the entire screen is refreshed every time a display is triggered.

[0070] like Figure 1 As shown in the figure, the voice interaction and data transmission process between the device and the cloud server in this embodiment is as follows: 1. Voice input and wake-up: The user speaks the wake-up word and target sentence to the device (such as "I want to write the word 'ancestor' of the motherland").

[0071] 2. Local endpoint detection: After receiving the sound, the device first performs wake word detection; then it starts voice activity detection (VAD) to determine the start and end of the voice locally.

[0072] 3. Audio Stream Upload: When the VAD detects the start of voice, the device sends a listen_start signal to the gateway via its wireless module, and then uploads the acquired PCM (Pulse Code Modulation) audio frame data stream in real time. The gateway forwards the data to the server.

[0073] 4. Parsing and Generation: When VAD determines that the speech has ended, the device sends a stop listening signal (listen_stop). The server first performs Automatic Speech Recognition (ASR) to extract the target text; then, it parses the text based on a preset rule template and generates corresponding writing guidance data.

[0074] 5. Result Distribution and Display: The server sends the generated writing guidance data back to the device. After receiving the data, the display control module decodes it and finally overlays it on the display device.

[0075] In other embodiments, Voice Activity Detection (VAD) can also be performed on a cloud server. The specific process is as follows: ① After being woken up, the device uploads the collected PCM audio frame data stream to the cloud server in real time; ②The cloud server performs real-time voice activity detection on the received audio stream to determine the start and end of the voice; ③ When the cloud-based VAD detects that continuous silence has reached the preset duration (600~1200ms), it determines that the voice has ended; ④ The cloud server performs speech recognition and target text parsing on the received complete audio.

[0076] This embodiment is suitable for scenarios where the device has limited computing power or where it is desirable to simplify the device logic.

[0077] To achieve flexible and natural stroke-by-stroke display control, this embodiment also provides a state machine model, combined with various interactive triggering methods, to achieve fine-grained control over the display process, as detailed below: 1. Definition of a state machine This embodiment preferably uses the following state transition model, and the definitions of each state are shown in Table 1: Table 1 State Definition Table

[0078] The basic path of state transition is: IDLE → LISTENING → UPLOADING → RENDER_READY. Then, depending on the user's operation or system settings, it enters sub-states such as FULL_VIEW, GUIDE_VIEW, STROKE_STEP_VIEW, or AUTO_PLAY.

[0079] 2. Interaction Triggering Method This embodiment supports the following three methods to trigger the display of stroke-by-stroke drawing or related operations: Key-triggered operation: Users operate via physical keys. Pressing a key once switches the system from the current state to the next stroke display state (e.g., jumping from the current stroke in STROKE_STEP_VIEW to the next stroke); pressing and holding a key can trigger a re-recording (state reverts to LISTENING) or return to the full character view (state jumps to FULL_VIEW).

[0080] Voice triggering: Users control the system via voice commands. For example, a user can say "next stroke," "previous stroke," "complete character," or "start from the beginning." The system will then parse the command via the cloud and execute the corresponding status transition based on the command content.

[0081] Automatic playback: After the user selects the automatic playback mode, the system enters the AUTO_PLAY state and automatically updates the screen display frame by frame according to the preset time interval (adjustable) to complete the automatic demonstration of the stroke animation.

[0082] By combining the aforementioned state machine with multimodal interactive triggering methods, refined and user-friendly control over the stroke-by-stroke display process is achieved. Users can flexibly choose between button, voice, or automatic playback modes for writing practice based on their learning progress and preferences, thereby improving user experience and learning efficiency.

[0083] Furthermore, this embodiment also provides a text display system based on speech semantic parsing, comprising: The voice acquisition and endpoint detection module is used to receive user voice commands and detect voice activity. The acquisition ends when continuous silence is detected for 600~1200ms. The voice acquisition and endpoint detection module supports local endpoint detection or streaming to the cloud for endpoint detection. The parsing and data generation module is used to recognize the collected speech, parse the target text based on the preset rule template, and generate the stroke data and guidance layer data of the target text. The device-side display control module is used to receive stroke-by-stroke drawing data and guide layer data, overlay them on the display device, and trigger the display stroke by stroke.

[0084] Furthermore, this embodiment also provides a target text writing practice device, including: The main controller, as the core processing unit, preferably uses a low-power ESP32 series microcontroller, which is responsible for coordinating the work of each module, running application logic, handling network communication, and driving the display screen.

[0085] Memory, including Flash memory and RAM, is used to store device firmware, applications, received calligraphy data, and temporary data during runtime.

[0086] A microphone is used to capture user voice commands.

[0087] The speaker module is used to broadcast the pronunciation annotations, operation prompts, or stroke-guided speech of the target text.

[0088] The display device is used to display calligraphy templates and stroke guidance information, preferably an electronic ink screen, and its driver chip model is preferably ST8899.

[0089] Wireless communication modules, typically integrated into the main controller (such as the Wi-Fi module in the ESP32), are used for data exchange with cloud servers.

[0090] Physical buttons, at least one in number, are used to perform different operation functions.

[0091] The main controller communicates with the memory, microphone (via analog-to-digital converter), display device driver chip and wireless communication module through internal bus or dedicated interface, forming a complete working system.

[0092] In this embodiment, the main controller connects to a microphone via I2S for audio acquisition and playback, connects to an e-ink screen driver chip via an SPI interface, and connects to an external Flash memory via an SPI interface.

[0093] Other preferred embodiments: 1. Local generation of the bootstrap layer: To reduce reliance on cloud services or enable offline functionality, the guidance layer data can also be generated locally on the device. After receiving the target text, the device can invoke a built-in lightweight dot matrix dithering algorithm to generate the guidance layer based on the local font library.

[0094] 2. Supports multiple languages: The method provided by this invention is also applicable to English letters, numbers, or symbols. For English letters, the cloud server queries their standard stroke order data, and the process of generating stroke-by-stroke display data is similar to that of Chinese character strokes.

[0095] 3. Data format extension: The stroke-by-stroke data is not limited to a strict 1-bit-per-pixel format. Any low-bit-depth bitmap format (such as 2-bit-per-pixel or 4-bit-per-pixel) that is compatible with the display characteristics of e-ink screens can be used, and their generation and display principles are the same.

[0096] 4. Vector path data format: In other embodiments, the stroke-by-stroke data is not limited to the 1bpp bitmap frame format, but can also be transmitted and displayed using a vector path data format (such as SVG format). The specific process is as follows: The cloud server generates vector path data for the target text, including the vector outlines or writing trajectories of each stroke; the vector path data is then sent to the device; the device renders and displays the text locally based on the received vector path data.

[0097] 5. Compatibility with multi-color display devices: In other embodiments, the method provided by the present invention is also applicable to multi-color display devices, such as multi-color e-ink screens. When the display device supports multiple colors, the stroke data can adopt a bitmap format corresponding to the color depth (such as 2bpp, 4bpp, or 8bpp) to support guided display of more colors.

[0098] 6. Other low-power display devices: In other embodiments, the display device is not limited to an e-ink screen, but can also be a thermal printer, e-paper, or other low-power display device. For a thermal printer, the device sends the received bitmap frame data directly to the print head for line-by-line printing, thereby generating a paper-based calligraphy template.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the design of this case. All equivalent changes made based on the key design of this case shall fall within the protection scope of this case.

Claims

1. A method for displaying text stroke by stroke based on speech and semantic parsing, characterized in that: Includes the following steps: S1: Receive user voice commands, perform voice activity detection on the voice commands, and end voice acquisition when continuous silence is detected for a preset duration; Among them, voice activity detection is performed on the device and / or cloud server, with a preset duration of 600~1200ms; S2: Perform semantic parsing on the collected speech text, and extract the target text from the speech text based on the preset rule template; S3: Provide the speech segment, recognized text, or target text to the writing guidance data generation module, and obtain the writing guidance data corresponding to the target text; The writing guidance data includes guidance layer data and stroke-by-stroke data, and the writing guidance data is one or more of the following: bitmap data, SVG path data, vector stroke data, center line trajectory data, and compressed frame data. The writing guidance data is generated through one or more of the following methods: character library retrieval, font parsing, vector path conversion, or artificial intelligence model generation. S4: On the display device, the background guide text and auxiliary grid lines are displayed based on the guide layer data, and the current stroke is displayed by overlaying the stroke data stroke by stroke.

2. The method for displaying text stroke by stroke based on speech semantic parsing as described in claim 1, characterized in that: The target text includes Chinese characters, letters, numbers, or symbols.

3. The method for displaying text stroke by stroke based on speech semantic parsing as described in claim 1, characterized in that: In step S1, the speech activity detection employs an energy thresholding method, the parameters of which include: The frame length is 20~30ms, the threshold for starting to speak is -45dBFS~-35dBFS, and the threshold for ending silence is -50dBFS~-40dBFS.

4. The method for displaying text stroke by stroke based on speech semantic parsing as described in claim 1, characterized in that: In step S2, the rule template includes a sentence structure for extracting target text from natural language.

5. The method for displaying text stroke by stroke based on speech semantic parsing as described in claim 1, characterized in that: In step S4, the triggering method for displaying each item includes at least one of the following: receiving physical button signals, receiving voice commands, or automatically switching at preset time intervals.

6. The method for displaying text stroke by stroke based on speech semantic parsing as described in claim 1, characterized in that: The bootstrap data is generated locally by the cloud server or the device.

7. The method for displaying text stroke by stroke based on speech semantic parsing as described in claim 1, characterized in that: The stroke-by-stroke data is a low-bit-depth bitmap format adapted for display devices; the display devices include electronic ink screens, thermal printers, or electronic paper.

8. The method for displaying text stroke by stroke based on speech semantic parsing as described in claim 1, characterized in that: Step S4 also includes displaying the phonetic annotation of the target text; The stroke-by-stroke data is a 1-bit-per-pixel bitmap frame, and the guide layer data is data generated by dot matrix dithering to simulate grayscale on a 1-bit-per-pixel display device.

9. A text display system based on speech semantic parsing, characterized in that: include: The voice acquisition and endpoint detection module is used to receive user voice commands and detect voice activity. The acquisition ends when continuous silence is detected for 600~1200ms. The voice acquisition and endpoint detection module supports local endpoint detection or streaming to the cloud for endpoint detection. The parsing and data generation module is used to recognize the collected speech, parse the target text based on the preset rule template, and generate the stroke data and guidance layer data of the target text. The device-side display control module is used to receive stroke-by-stroke drawing data and guide layer data, overlay them on the display device, and trigger the display stroke by stroke.

10. A target character writing practice device, characterized in that: It includes a main controller, memory, microphone, display device, speaker module, and wireless communication module; The main controller is communicatively connected to the memory, microphone, display device, speaker module, and wireless communication module, respectively. The memory stores a computer program, which, when executed by the main controller, implements the text display method based on speech semantic parsing as described in any one of claims 1 to 8.

Images