A digital mural creation system integrating voice control

Abstract

This invention relates to the field of multimedia data technology, specifically to a voice-controlled digital mural creation system. The system includes: a voice command parsing module, a creative intent recognition module, a graphic element generation module, and a command mapping self-learning module. In this invention, by combining word segmentation and category label reasoning, clear identification of semantic structure and effective extraction of creative intent can be achieved. Furthermore, through the generation of control points for vector paths and the processing of brush parameter combinations, the automatic drawing process of graphic elements is completed. Simultaneously, continuous monitoring of subsequent user corrections and updates to mapping relationships are introduced, realizing dynamic optimization of lexical knowledge mapping rules. This improves the system's adaptability and response efficiency when handling complex semantic expressions and diverse graphic needs, enhances the interactivity and intelligence of digital mural creation, and promotes the continuity and accuracy of creative content from voice commands to visual presentation.

Description

Technical Field

[0001] This invention relates to the field of multimedia data technology, and in particular to a digital mural creation system that integrates voice control. Background Technology

[0002] The field of multimedia data technology involves the processing, management, and presentation of multimodal information in an electronic digital format. Its core aspects include the acquisition, editing, storage, and organization of data such as images, audio, video, and text, as well as the integration and application of different types of data within computer systems. Overall, this technology encompasses a complete system from the generation and classification of multimedia content to the structured management and interactive presentation of data, supporting the realization of various scenarios such as cultural dissemination, digital art, and visual display. Among these, traditional mural digital creation systems refer to the generation and combination of mural images using image editing software and multimedia processing tools. The technical aspects addressed are mural creation and content combination under digital conditions. Typically, this involves acquiring source images using image acquisition devices and then generating digital murals on a computer through image drawing, editing, and multimedia data fusion.

[0003] Existing technologies rely on image acquisition and editing to integrate materials and create images, lacking a mechanism for processing natural language expressions. The process requires manual execution of image retrieval, editing, and combination steps, resulting in a fragmented and inefficient creative workflow. This is especially true in scenarios requiring frequent adjustments to composition or rapid generation of graphic content, where there are issues of lag and insufficient content adaptability. Furthermore, when faced with ambiguous or frequently changing subjective intentions from users, existing technologies cannot provide knowledge feedback or adjust instructions based on user behavior, limiting the system's ability to learn and adapt to changes in creative needs. This affects the scalability and precision of digital mural creation in diverse expressive environments. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a digital mural creation system that integrates voice control.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a digital mural creation system integrating voice control, comprising: a voice command parsing module, which acquires audio waveform data, separates voice segments in the audio waveform data, parses out text strings carrying timestamps, and transmits the text strings carrying timestamps to a creative intent recognition module; The creative intent recognition module segments the text string, matches word units with the word knowledge base to assign category labels, and judges the category label sequence to generate structured creation instructions, and transmits the structured creation instructions to the graphic element generation module. The graphic element generation module parses the structured creation instructions and generates vector path control points. Then, it combines the brush parameter set to stroke the vector path control points, calculates the canvas element dataset, and passes the canvas element dataset to the instruction mapping self-learning module. The instruction mapping self-learning module monitors the correction instructions of the canvas element dataset, records the initial instructions and correction parameters, analyzes the co-occurrence patterns, and then adjusts the mapping rules of the lexical knowledge base according to the co-occurrence patterns.

[0006] As a further aspect of the present invention, the text string carrying the timestamp specifically comprises a translated word sequence and timestamp information; The structured creation instructions include creation object words, painting action words, attribute description words, spatial composition words, and adjustment and correction words; The canvas element dataset specifically refers to vector path data, brush parameters, and color information; The mapping rules specifically consist of a set of instruction tuples and corresponding operation parameters.

[0007] As a further embodiment of the present invention, the voice command parsing module includes: an audio data acquisition submodule, which acquires the audio waveform data and performs preprocessing to generate an original audio stream; The voice activity detection submodule calculates the short-time energy and zero-crossing rate of each audio frame based on the original audio stream, determines the start and end points of voice activity according to preset energy thresholds and zero-crossing rate thresholds, and separates the voice segments from the original audio stream to obtain independent voice data. The speech content translation submodule calls the acoustic model and language model to recognize the independent speech data and outputs a translated word sequence without time information; The timestamp alignment submodule integrates the start and end points of the speech activity and the translated word sequence, and assigns a start time and duration to each word unit in the translated word sequence through a forced alignment algorithm to generate the text string carrying the timestamp. The preset energy threshold includes a high energy threshold and a low energy threshold. The high energy threshold is used to initially determine the existence of the speech core region, and the low energy threshold is used to define the start and end boundaries of the speech segment. The zero-crossing rate threshold includes a high zero-crossing rate threshold and a low zero-crossing rate threshold, which are used to help distinguish between voiceless and voiced components in speech.

[0008] As a further embodiment of the present invention, the creative intent recognition module includes: a text string segmentation submodule, which obtains the text string carrying the timestamp, and segments it into multiple word units containing independent semantics according to the maximum matching method and a preset dictionary, and generates a segmentation result set; The word category matching submodule traverses each word unit in the word segmentation result set, queries the word knowledge base to match and assign a category label from the following categories: creative object, painting action, attribute description, spatial composition, or adjustment and correction, and generates a label sequence. The syntax structure analysis submodule analyzes the permutations and combinations of the tag sequences, matches the tag sequences using a predefined authoring instruction syntax tree, identifies the instruction structure pattern, and obtains the instruction template. The creation instruction generation submodule, based on the instruction template, fills the word units carried in the tag sequence into the corresponding parameter slots of the instruction template to generate the structured creation instruction.

[0009] As a further embodiment of the present invention, the graphic element generation module includes: an instruction content parsing submodule, which receives the structured creation instruction and extracts key parameters such as the type of creation object, spatial position, color and brush style from it to generate a standardized parameter list; The control point positioning submodule calls the corresponding geometric primitive model according to the creation object type in the standardized parameter list, and calculates a set of initial vector path control points in the canvas coordinate system in combination with the spatial position to obtain the control point coordinate set. The path curve fitting submodule uses a cubic Bézier curve algorithm to perform interpolation and smoothing on the control point coordinate set, generating a smooth vector path connecting all vector path control points and obtaining vector path data. The brush effect rendering submodule calls the parameters in the brush parameter set that match the brush style in the standardized parameter list to perform outlining and filling rendering on the vector path data, calculates the rendered pixel information, and integrates the vector path data, the selected brush parameters and color information to generate the canvas element dataset.

[0010] As a further embodiment of the present invention, the instruction mapping self-learning module includes: a user correction monitoring submodule, which continuously monitors subsequent manual adjustment operations for the canvas element dataset after the dataset is generated, and when an adjustment operation is detected, records the adjustment correction terminology and specific correction parameters corresponding to the operation, and generates a correction event record. The instruction association analysis submodule traces back to the most recently generated canvas element dataset before the correction event occurred, and indexes the structured creation instruction that generated the dataset, establishing the association between the initial instruction, canvas elements, and correction operation to obtain the instruction correction pair; The co-occurrence pattern mining submodule statistically analyzes multiple stored instruction correction pairs, uses an association rule algorithm to calculate the co-occurrence frequency and confidence between different initial instructions and correction parameters, filters out association patterns that are higher than a preset confidence threshold, and generates the co-occurrence pattern. The knowledge base rule update submodule locates the mapping rule related to the initial instruction in the lexical knowledge base based on the co-occurrence pattern, and adjusts the default output parameter of the mapping rule according to the correction parameter to generate the updated lexical knowledge base.

[0011] As a further aspect of the present invention, when the syntax structure analysis submodule identifies the instruction structure pattern, it specifically operates by generating multiple candidate instruction structure patterns for the tag sequence. The instruction confidence score for each candidate instruction structure pattern is calculated using the following formula. : ; in, The instruction confidence level represents the candidate instruction structure pattern. This represents the word unit at the i-th position in the label sequence. The representative assigns the term unit Category tags, This represents the word unit recorded in the lexical knowledge base. Belongs to the category label The prior probability, Represents all category labels The complete category label sequence constitutes This represents the structural probability that the complete category label sequence conforms to predefined grammar rules. Represents the weighting coefficient. The counting variable is the summation symbol, and its range is from 1 to n. The length of the label sequence; Finally, the command confidence level is selected. The highest-ranking candidate instruction structure pattern is selected as the instruction structure pattern, and the instruction template is generated.

[0012] As a further aspect of the present invention, when the control point positioning submodule calculates the initial vector path control point, if the standardized parameter list contains spatial composition terms, its specific operation is as follows: obtain the canvas bounding box coordinates of the reference object specified in the structured creation instruction; Based on the semantics of the spatial composition lexical units, a target generation region is calculated based on the canvas bounding box coordinates of the reference object; Invoke the geometric primitive model corresponding to the type of the creation object. This model pre-determines the number and relative position distribution of control points required to describe the outline of the object. The relative position distribution of the model is linearly mapped to the target generation area, and adjusted in combination with random offsets. The absolute coordinates of each control point in the canvas coordinate system are calculated to generate the control point coordinate set. The absolute coordinates of each control point in the canvas coordinate system are calculated using the following formula: ; in, This represents the final absolute coordinates of the j-th control point. This represents the coordinates of the upper left corner reference point of the target generation area. This represents the width of the target generation region. This represents the height of the target generation region. This represents the normalized relative position coordinates of the j-th control point obtained from the geometric primitive model, and its value ranges from 0 to 1. The perturbation coefficient represents the control of the random offset amplitude. Represents a standard normally distributed random number. The index used to identify control points.

[0013] As a further aspect of the present invention, when the knowledge base rule update submodule adjusts the mapping rule, the specific operation is as follows: locate the specific operation parameter in the mapping rule that directly corresponds to the correction parameter; The weight values ​​of the specific operation parameters are updated using the following formula. : ; in, This represents the updated weight value. This represents the weight value of the specific operation parameter before the update. Represents the learning rate parameter. This represents the number of corrections to the mapping rule observed within a statistical period. This represents the target parameter value determined by the user after the k-th correction. This represents the initial parameter value generated by the mapping rule before the k-th correction. The counting variable is for the summation sign, and its range is from 1 to N; The calculated updated weight values Replace the original The updated lexical knowledge base is generated.

[0014] As a further aspect of the present invention, when the voice activity detection submodule separates the voice segments, the specific operation is as follows: the acquired original audio stream is divided into continuous audio frames of fixed duration; For each audio frame, calculate its short-time energy and short-time average zero-crossing rate, and generate an energy parameter sequence and a zero-crossing rate parameter sequence. By setting high-energy threshold, low-energy threshold, high zero-crossing rate threshold and low zero-crossing rate threshold, the start and end points of speech can be preliminarily identified by comparing the energy parameter sequence with the zero-crossing rate parameter sequence. A dual-threshold algorithm is used to verify the determined speech start and end points. Specifically, the speech core area determined by the high threshold is extended to both sides until it is below the low threshold to determine the boundary of the speech segment. Waveform data within the boundary is then extracted from the original audio stream to obtain the independent speech data.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, by parsing and timestamping voice data, the acquisition of instruction information becomes more timely and accurate. Combined with word segmentation and category label reasoning, it is possible to achieve clear identification of semantic structure and effective extraction of creative intent. Furthermore, by generating control points for vector paths and processing brush parameter combinations, the automatic drawing process of graphic elements is completed. At the same time, the continuous monitoring of subsequent user correction operations and the updating of mapping relationships are introduced, realizing the dynamic optimization of word knowledge mapping rules. This improves the system's adaptability and response efficiency when dealing with complex semantic expressions and diverse graphic needs, enhances the interactivity and intelligence of digital mural creation, and promotes the continuity and accuracy of creative content from voice commands to visual presentation. Attached Figure Description

[0016] Figure 1 This is a structural block diagram of the digital mural creation system integrating voice control according to the present invention; Figure 2 This is a flowchart of the voice command parsing process of the present invention; Figure 3 This is a flowchart illustrating the creative intent recognition process for this invention. Figure 4 This is a flowchart illustrating the process of generating graphic elements in this invention. Figure 5 This is a flowchart illustrating the instruction mapping self-learning process of this invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the software-based technical solution is described in detail below with reference to system architecture diagrams and embodiments. It should be understood that the specific embodiments described herein are only for explaining the technical solutions of this invention and do not constitute a limitation on the scope of protection.

[0018] In the description of this invention, the system architecture relationships or data processing flows indicated by terms such as "layer," "module," "interface," "data flow," "client," and "server" are all defined based on the architecture diagram or flowchart corresponding to the embodiments. This way of describing is only used to clearly illustrate the logical relationships between the elements in the technical solution, and not to limit the physical deployment form. The term "multiple" includes two or more technical units, including but not limited to multiple data nodes, processing threads, service instances, or functional components and other scalable elements. The specific number is determined according to the actual business scenario and needs to be specifically specified.

[0019] Please see Figure 1 and Figure 2 This invention provides a technical solution: a digital mural creation system integrating voice control, comprising: The voice command parsing module acquires audio waveform data, separates the voice segments in the audio waveform data, and parses out the text string carrying the timestamp, and then passes the text string carrying the timestamp to the creative intent recognition module. The text string carrying a timestamp specifically consists of a translated word sequence and timestamp information; The voice command parsing module includes: an audio data acquisition submodule, which acquires audio waveform data and performs preprocessing to generate the raw audio stream; The speech activity detection submodule calculates the short-time energy and zero-crossing rate of each audio frame based on the original audio stream, determines the start and end points of speech activity according to the preset energy threshold and zero-crossing rate threshold, and separates speech segments from the original audio stream to obtain independent speech data. When separating speech segments, the speech activity detection submodule specifically operates by dividing the acquired raw audio stream into continuous audio frames of fixed duration. For each audio frame, calculate its short-time energy and short-time average zero-crossing rate, and generate an energy parameter sequence and a zero-crossing rate parameter sequence. By setting high-energy threshold, low-energy threshold, high zero-crossing rate threshold and low zero-crossing rate threshold, the speech start point and end point are initially identified by comparing the energy parameter sequence and the zero-crossing rate parameter sequence. A dual-threshold algorithm is used to verify the determined start and end points of speech. That is, the speech core area determined by the high threshold is extended to both sides until it is below the low threshold to determine the boundary of the speech segment. The waveform data within the boundary is extracted from the original audio stream to obtain independent speech data. The speech content translation submodule calls the acoustic model and language model to recognize independent speech data and outputs a translated word sequence without time information; The timestamp alignment submodule integrates the start and end points of the speech activity and the translated word sequence. It assigns a start time and duration to each word unit in the translated word sequence through a forced alignment algorithm, generating a text string carrying a timestamp. The preset energy thresholds include a high energy threshold and a low energy threshold. The high energy threshold is used to initially determine the existence of the speech core region, while the low energy threshold is used to define the start and end boundaries of a speech segment. The zero-crossing rate threshold includes a high zero-crossing rate threshold and a low zero-crossing rate threshold. The high zero-crossing rate threshold and the low zero-crossing rate threshold are used to help distinguish between voiceless and voiced components in speech.

[0020] The audio data acquisition submodule is configured with a sampling frequency of 16kHz and a sampling bit depth of 16bit in mono mode to capture ambient sound. The acquired raw audio waveform data is sequentially read into a first-in-first-out (FIFO) buffer with a size of 2048 samples, and the data is processed in a streaming manner. In the preprocessing step, DC component cancellation is first performed on each frame of data, specifically by subtracting the average value of all samples in that frame from the frame's values. Subsequently, the signal is processed through a pre-emphasis filter, the transfer function of which is... For example, if an audio clip is acquired, and the amplitudes of its first 5 sampling points are {50, 55, 48, 52, 60}, and the average value of the clip is 53, then the amplitudes of the first 5 sampling points after DC component removal are updated to [-3, 2, -5, -1, 7]. During pre-emphasis processing, the new value of the second sampling point is... The processed data forms the raw audio stream, which is then passed to the speech activity detection submodule.

[0021] After receiving the raw audio stream, the voice activity detection submodule divides it into continuous, fixed-duration audio frames. The frame length is set to 25 milliseconds (ms), which is 400 sampling points (16000Hz * 0.025s); the frame shift is set to 10 milliseconds (ms), which is 160 sampling points, meaning there is a 15-millisecond overlap between adjacent frames. For each audio frame, its short-time energy (STE) and short-time zero-crossing rate (ZCR) are calculated. The short-time energy is obtained by calculating the sum of the squares of the amplitudes of all sampling points within the frame. For example, for a frame containing 400 sampling points... The audio frame with an STE value of The short-time average zero-crossing rate (ZCR) is obtained by calculating the number of sign changes between adjacent sampling points within a frame and then dividing by the frame length minus one. For example, for the same frame, its ZCR value is... ,in This is a sign function. Therefore, the original audio stream is converted into a sequence of energy parameters and a sequence of zero-crossing rate parameters.

[0022] To accurately distinguish between speech and non-speech segments, energy thresholds and zero-crossing rate thresholds need to be set. These thresholds are set based on statistical analysis of a large number of background noise samples and speech samples. The tests were conducted on a calibration dataset containing 500 segments of pure background noise (such as indoor fan noise or keyboard typing) and 500 segments of clean speech.

[0023] Table 1. Statistical table of threshold calibration experimental data; As shown in Table 1, through analysis of the calibration dataset, the high-energy threshold ( The value is set to be at a lower position in the speech energy distribution, and significantly higher than the energy value of most noise. Specifically, it is set to the geometric mean of the maximum background noise energy and the average speech energy. Low energy threshold ( The value is set slightly higher than the 95th percentile of the background noise level to effectively filter out most of the noise. Specifically, it is set to... High zero-crossing rate threshold ( ) and low zero-crossing rate threshold ( This is used to assist in the identification of voiceless consonants. Based on the characteristic that voiceless consonants (such as 's', 'sh') have high ZCR (zoom retrieval rate) and voiced consonants (such as vowels 'a', 'o') have low ZCR, Set to 0.35. Set to 0.18.

[0024] When recognizing speech segments, the energy parameter sequence is first traversed to find energy values ​​that exceed a certain threshold for N consecutive frames (e.g., N=3). The specified interval is marked as the speech core region. Then, a dual-threshold algorithm is used to verify the determined speech start and end points. Starting from the boundary of the speech core region, the algorithm searches forward and backward respectively until the first energy value below a certain threshold is found. The audio frames are defined, with the start and end positions of each frame defined as the boundaries of a speech segment. For example, if frames 105 to 115 are detected as the core speech region, the search begins from frame 105 and proceeds backwards, finding that the energy value of frame 101 is... lower than The energy value of frame 102 is... higher than Therefore, the start time point of frame 102 (101 * 10 ms = 1.01 s) is recorded as the speech start point. Similarly, the end point is determined by searching backward from frame 115. Waveform data within this boundary is extracted from the original audio stream to obtain independent speech data.

[0025] The speech content translation sub-module receives independent speech data and inputs it into a pre-trained acoustic model based on a deep neural network. This model maps audio features (such as MFCC) to a sequence of phoneme probabilities. Subsequently, an N-gram based language model combines these phoneme probabilities and uses the Viterbi decoding algorithm. With the sequence of phoneme probabilities as the observation sequence and the word transition probabilities provided by the pre-built N-gram language model as the state transition probabilities, in the decoding network composed of the dictionary, it calculates and traces back to obtain the word path with the maximum joint probability, and this path is the most likely word sequence. For example, for a speech data with the pronunciation of "draw a red apple", the acoustic model outputs a series of phoneme probabilities, and the language model determines that the probability of "red" followed by "of" and then "apple" is higher than other combinations, and finally determines the translated word sequence without time information: "draw a red apple".

[0026] The timestamp alignment sub-module integrates the start point (such as 1.01 seconds) and the end point (such as 3.51 seconds) of the speech activity and the translated word sequence. Through the forced alignment algorithm, each word unit in the word sequence is matched with a specific paragraph in the original speech data. This algorithm uses the acoustic model to calculate the acoustic score of the word sequence within a given time period and finds the best alignment path through dynamic programming, assigning the start time and duration to each word unit in the translated word sequence. For example, "draw" [1.25s, 0.20s], "a" [1.45s, 0.30s], "red" [1.75s, 0.40s], "of" [2.15s, 0.15s], "apple" [2.30s, 0.50s]. Finally, a text string with timestamps is generated, and its data structure is: {text: ["draw", "a", "red", "of", "apple"], timestamps: [{start: 1.25, duration: 0.20}, {start: 1.45, duration: 0.30}, {start: 1.75, duration: 0.40}, {start: 2.15, duration: 0.15}, {start: 2.30, duration: 0.50}]}.

[0027] Please refer to Figure 1 and Figure 3 the creative intention recognition module, which segments the text string, matches the word units with the lemma knowledge base to assign category labels, and judges the category label sequence to generate structured creative instructions, and passes the structured creative instructions to the graphic element generation module; The structured creative instructions include creative object lemmas, painting action lemmas, attribute description lemmas, spatial composition lemmas, adjustment and correction lemmas; The creative intent recognition module includes: a text string segmentation submodule, which obtains a text string carrying a timestamp, and segments it into multiple word units containing independent semantics based on the maximum matching method and a pre-set dictionary, generating a segmentation result set; The lexical category matching submodule iterates through each word unit in the word segmentation result set, queries the lexical knowledge base to match and assign a category label from the following categories: creative object, painting action, attribute description, spatial composition, or adjustment and correction, and generates a label sequence. The syntax structure analysis submodule analyzes the permutations and combinations of tag sequences, matches the tag sequences using a predefined authoring instruction syntax tree, identifies the instruction structure pattern, and obtains the instruction template. When identifying instruction structure patterns, the syntax structure analysis submodule specifically performs the following operation: for the tag sequence, it generates multiple candidate instruction structure patterns. The instruction confidence score for each candidate instruction structure pattern is calculated using the following formula. : ; in, Instruction confidence represents the candidate instruction structure pattern. This represents the word unit at the i-th position in the label sequence. Representatives endow words with units Category tags, Represents the word units recorded in the lexical knowledge base. Category tags The prior probability, Represents all category labels The complete category label sequence constitutes The structural probability that the complete category label sequence conforms to predefined grammatical rules. Represents the weighting coefficient. The counting variable is the summation symbol, and its range is from 1 to n. The length of the label sequence; Finally, select the instruction confidence level. The highest-ranking candidate instruction structure pattern is selected as the instruction structure pattern, and an instruction template is generated. The creation instruction generation submodule, based on the instruction template, fills the word units carried in the tag sequence into the corresponding parameter slots of the instruction template to generate structured creation instructions.

[0028] Text string tokenization sub-module, which obtains a text string with timestamps, e.g., {text: ["On", "the canvas", "on the left", "draw", "a", "red", "apple"], timestamps: [... ]}. According to the maximum matching method and a pre-set domain dictionary, which contains common words in the creation field, such as "on the left of the canvas", "red", "apple", it is segmented into multiple word units with independent semantics. The tokenization process proceeds from left to right, looking for the longest matching word in the dictionary. For the example string, first "on" is matched, then "the canvas", but there is a longer word "on the left of the canvas" in the dictionary. Due to the maximum matching principle, the system will not separate "the canvas" and "on the left". Therefore, the tokenization result set is: ["on the left of the canvas", "draw", "a", "red", "apple"].

[0029] Lemma category matching sub-module, which traverses each word unit in the tokenization result set and queries a pre-constructed lemma knowledge base. This knowledge base stores the mapping relationships and prior probabilities between word units and five category labels (creation object, painting action, attribute description, spatial composition, adjustment and correction). For example, for the word unit "apple", the mapping recorded in the knowledge base is: {"creation object": 0.95, "attribute description": 0.01,...}. For "on the left of the canvas", its mapping is: {"spatial composition": 0.98,...}. For "draw", its mapping is: {"painting action": 0.99,...}. For "red", its mapping is: {"attribute description": 0.97,...}. "A" and "of" are identified as stop words at this stage because their semantics have a low correlation with the creation instructions and are not assigned main category labels. After traversal, a label sequence corresponding to the tokenization result set is generated, e.g., [spatial composition, painting action, attribute description, creation object].

[0030] Syntax structure analysis sub-module, which analyzes the permutations and combinations of this label sequence. For the label sequence [spatial composition, painting action, attribute description, creation object], the system generates multiple candidate instruction structure patterns. For example, candidate pattern 1 (C_seq1) is: [spatial composition] -> [painting action] -> [attribute description] -> [creation object]. Candidate pattern 2 (C_seq2) is: [spatial composition] -> [painting action] -> [creation object] -> [attribute description] (this is ungrammatical in syntax but is a candidate). Calculate the instruction confidence for each candidate pattern .

[0031] The formula is .

[0032] Where represents the final confidence score of the candidate instruction structure pattern; It is a weighting coefficient used to balance the accuracy of word classification with the rationality of grammatical structure; Represents the total number of word units in the label sequence; the subscript in the summation symbol. Represents the position index of the word unit in the label sequence, traversing from 1 to... ; Represents the first label in the label sequence Word units in each position; Representatives endow words with units Category tags; Representative words Category The prior probability retrieved from the lexical knowledge base; Represents all category labels The complete category label sequence constitutes; This represents the structural probability that the tag sequence conforms to a predefined grammar rule; this value is obtained from the grammar rule library.

[0033] The setup process is as follows: Prepare a test instruction set containing 100 correctly labeled word categories and grammatical structures. The test set is evaluated using a step size of 0.1 to 0.9, with each step increasing by 0.1. Command recognition is performed, and the values ​​for each command are calculated. The recognition accuracy under the given values. Experiments showed that when When the value is 0.6, the system achieves the highest overall recognition accuracy among word classification errors and grammatical structure errors, reaching a maximum overall recognition accuracy of 96%. Therefore, in this embodiment... The value is 0.6.

[0034] Example: For candidate pattern 1: =[Spatial composition, painting action, attribute description, creative object] word sequence =[“on the left side of the canvas”, “paint”, “red”, “apple”](stop words ignored) ; For candidate mode 2: =[Spatial Composition, Painting Action, Subject Matter, Attribute Description] The value is the same, 3.89. ; Compare the confidence levels of the two candidate patterns. Higher than This result indicates that candidate mode 1 has a higher confidence level. Therefore, the system selects the instruction confidence level. The highest candidate mode 1 is selected as the final instruction structure mode, and a corresponding instruction template is generated: spatial composition, painting, action attribute description, and creation object.

[0035] The creation instruction generation submodule, based on the selected instruction template, fills the corresponding parameter slots of the instruction template with the word units carried in the tag sequence. For example, "on the left side of the canvas" is filled into parameter 1, "draw" into parameter 2, "red" into parameter 3, and "apple" into parameter 4. The final generated structured creation instruction has the following data structure: {command: "create", object: "apple", attributes: {color: "red"}, location: "left side of the canvas", action: "draw"}.

[0036] Please see Figure 1 and Figure 4 The graphic element generation module parses the structured creation instructions and generates vector path control points. Then, it combines the brush parameter set to stroke the vector path control points, calculates the canvas element dataset, and passes the canvas element dataset to the instruction mapping self-learning module. The canvas element dataset specifically refers to vector path data, brush parameters, and color information; The graphic element generation module includes: an instruction content parsing submodule, which receives structured creation instructions and extracts key parameters such as the type of creation object, spatial position, color and brush style, and generates a standardized parameter list; The control point positioning submodule calls the corresponding geometric primitive model according to the creation object type in the standardized parameter list, and calculates a set of initial vector path control points in the canvas coordinate system based on the spatial position, thus obtaining the control point coordinate set. When calculating the initial vector path control points, if the standardized parameter list contains spatial composition terms, the control point positioning submodule will specifically operate as follows: obtain the canvas bounding box coordinates of the reference object specified in the structured creation instruction; Based on the semantics of spatial composition terms, a target generation region is calculated based on the canvas bounding box coordinates of the reference object; Call the geometric primitive model corresponding to the type of the created object. This model pre-determines the number and relative position distribution of control points required to describe the outline of the object. The relative position distribution of the model is linearly mapped to the target generation area, and adjusted in combination with random offsets. The absolute coordinates of each control point in the canvas coordinate system are calculated to generate a control point coordinate set. The absolute coordinates of each control point in the canvas coordinate system are calculated using the following formula: ; in, This represents the final absolute coordinates of the j-th control point. This represents the coordinates of the top-left corner reference point of the target generation area. Represents the width of the target generation region. Represents the height of the target generation region. This represents the normalized relative position coordinates of the j-th control point obtained from the geometric primitive model, with values ​​ranging from 0 to 1. The perturbation coefficient represents the control of the random offset amplitude. Represents a standard normally distributed random number. Index for identifying control points; The path curve fitting submodule uses a cubic Bézier curve algorithm to perform interpolation and smoothing on the control point coordinate set, generating a smooth vector path connecting all vector path control points and obtaining vector path data. The brush effect rendering submodule calls the parameters that match the brush style in the brush parameter set and the standardized parameter list to perform stroke and fill rendering on the vector path data, calculates the rendered pixel information, and integrates the vector path data, the selected brush parameters and color information to generate a canvas element dataset.

[0037] The instruction parsing submodule receives the structured creation instruction {command:“create”,object:“apple”,attributes:{color:“red”},location:“left side of the canvas”,action:“draw”}. The submodule extracts the creation object type as “apple”, the spatial location as “left side of the canvas”, the color as “red”, and the brush style as the system default “standard circular” if not specified here. These key parameters are organized into a standardized parameter list, for example: {object_type:'apple',location_constraint:'left_canvas',color_value:'#FF0000',brush_style:'standard_circle'}.

[0038] The control point localization submodule, based on the "apple" creation object type in the standardized parameter list, calls the corresponding geometric primitive model. This model is a pre-defined data structure that stores the number of control points required to describe the apple's outline and their normalized relative position distribution. For example, the apple model pre-defined has 5 control points, and its normalized relative position coordinate set is as follows: The standardized parameter list includes the spatial composition term "left side of the canvas," and its operation is as follows: First, obtain the total size of the canvas, for example, 1920x1080 pixels. Based on the semantics of "left side of the canvas," calculate the target generation area. In this example, "left side" is defined as the first third of the horizontal area of ​​the canvas, i.e., the x-coordinate range is [0, 640], and the y-coordinate range is [0, 1080]. Therefore, the coordinates of the top-left corner reference point of the target generation area are... ,width ,high Subsequently, the relative positional distribution of the geometric primitive model is linearly mapped to the target generation area, and adjusted in conjunction with random offsets to calculate the absolute coordinates of each control point in the canvas coordinate system.

[0039] Absolute coordinates are obtained through the formula accomplish.

[0040] in, The index represents the final absolute coordinates of the j-th control point on the canvas. For the index of the control point; These are the coordinates of the top-left corner reference point of the target generation area; It is the width of the target generation region; It is the height of the target generation area; It is the x-component of the normalized relative position coordinates of the j-th control point obtained from the geometric primitive model; It is the y-component of the normalized relative position coordinates of the j-th control point obtained from the geometric primitive model; It is the disturbance coefficient that controls the amplitude of random offset; It is a standard normally distributed random number generator used to generate random numbers with a mean of 0 and a variance of 1.

[0041] Disturbance coefficient The settings were determined through a user preference study. Twenty test users were shown settings based on different... One hundred images were generated using values ​​(ranging from 0 to 20), and users were asked to rate the images for "naturalness" and "aesthetics." Statistical analysis showed that when... User ratings are highest when the value is between 4.0 and 6.0. Therefore, in this embodiment... The default value is set to 5.0.

[0042] Example (calculating the coordinates of the first two control points): Canvas size: 1920x1080. Target area: . Assuming two calls Generate random number pairs respectively and .

[0043] Calculate the first control point : ; Calculate the second control point : ; Similarly, the coordinate set of all 5 control points is calculated as follows: [(321.5,105.5),(572.0,433.0),(450.0,975.0),(189.0,970.0),(68.0,435.0)].

[0044] The path curve fitting submodule receives this set of control point coordinates. It then uses a cubic Bézier curve algorithm to interpolate and smooth these points, generating a smooth vector path connecting all the control points of the vector path, thus obtaining the vector path data.

[0045] The brush effect rendering submodule calls the parameters from the brush parameter set that match the brush style "standard_circle" in the normalized parameter list. This parameter set defines the brush diameter as 5 pixels, the hardness as 0.8, and the color as "#FF0000" (red) from the normalized parameter list. Then, it performs stroke and fill rendering on the vector path data generated in the previous step. The stroke operation draws a 5-pixel-wide red line along the path, and the fill operation sets all pixels within the closed area of ​​the path to red. The calculated rendered pixel information, along with the vector path data, the selected brush parameters, and the color information, are integrated into a canvas element dataset and passed to the instruction mapping self-learning module.

[0046] Please see Figure 1 and Figure 5 The instruction mapping is self-learning module, which monitors the correction instructions of the canvas element dataset, records the initial instructions and correction parameters, analyzes the co-occurrence patterns, and then adjusts the mapping rules of the lexical knowledge base according to the co-occurrence patterns. The mapping rules specifically consist of a set of instruction words and corresponding operation parameters; The instruction mapping self-learning module includes: a user correction monitoring submodule, which continuously monitors subsequent manual adjustment operations for the canvas element dataset after the dataset is generated. When an adjustment operation is detected, the adjustment correction terminology and specific correction parameters corresponding to the operation are recorded, and a correction event record is generated. The instruction association analysis submodule traces back to the most recently generated canvas element dataset before the correction event occurred, and indexes the structured creation instruction that generated the dataset. It establishes the association between the initial instruction, canvas elements, and correction operation to obtain the instruction correction pair. The co-occurrence pattern mining submodule statistically analyzes multiple stored instruction correction pairs, uses association rule algorithms to calculate the co-occurrence frequency and confidence between different initial instructions and correction parameters, filters out association patterns that are higher than the preset confidence threshold, and generates co-occurrence patterns. The knowledge base rule update submodule, based on the co-occurrence pattern, locates the mapping rule related to the initial instruction in the lexical knowledge base, and adjusts the default output parameter of the mapping rule according to the correction parameter to generate the updated lexical knowledge base; When adjusting mapping rules, the knowledge base rule update submodule performs the following operation: locates the specific operation parameter in the mapping rule that directly corresponds to the correction parameter; The weight values ​​of specific operation parameters are updated using the following formula. : ; in, This represents the updated weight value. This represents the weight value of the specific operation parameter before the update. Represents the learning rate parameter. This represents the number of corrections to the mapping rule observed within a statistical period. This represents the target parameter value determined by the user after the k-th correction. This represents the initial parameter value generated by the mapping rule before the k-th correction. The counting variable is for the summation sign, and its range is from 1 to N; The calculated updated weight values Replace the original This generates an updated lexical knowledge base.

[0047] The user-corrected monitoring submodule continuously monitors subsequent user actions on the element after a red apple has been created on the canvas. For example, the user enlarges the apple by 20% using a graphical interface tool. This action is captured by the submodule and identified as an adjustment / correction operation. The corresponding adjustment / correction term (in this example, the implicit "enlarge" operation) and the specific correction parameter, namely a scaling ratio of 1.2, are recorded. This information is formatted as a correction event record: {element_id:'apple_01',operation:'scale',parameters:{factor:1.2},timestamp:'2025-09-09T04:09:00Z'}.

[0048] The instruction association analysis submodule, based on the element_id 'apple_01' in the correction event log, traces back to find the original instruction that generated the element. The system indexes the most recent structured creation instruction that generated this element as: {command:“create”,object:“apple”,attributes:{color:“red”},location:“left side of the canvas”,action:“draw”}. The submodule associates this initial instruction, the generated canvas element, and the correction operation to establish an instruction-correction pair: ({initial_command:{…}},{correction:{operation:'scale',factor:1.2}}). This data pair is stored in a dedicated log database.

[0049] The co-occurrence pattern mining submodule sets an activation threshold, such as 100 entries. This submodule is activated when the number of instruction correction pairs stored in the log database reaches this threshold. It uses association rule algorithms (such as Apriori) to statistically analyze this data. The algorithm aims to discover patterns such as "If a user issues instruction X, they are very likely to execute correction Y." Specifically, it calculates the co-occurrence frequency and confidence between specific terms in the initial instruction (such as "apple") and correction parameters (such as "scale>1.0"). For example, analysis revealed that in 150 creation instructions containing "apple," 105 were followed by a scaling operation (scale>1.0). Its support is 105 / 150 = 70%. The system's preset confidence threshold is 60%. Since 70% > 60%, "create 'apple' -> scale up" is identified as a high-confidence association pattern, i.e., a co-occurrence pattern.

[0050] The knowledge base rule update submodule, based on the mined co-occurrence patterns, locates the mapping rules in the lexical knowledge base related to the initial instruction (creating "apple"). Specifically, it locates the default operation parameters corresponding to the lexical "apple," which includes a default size or scaling parameter. Based on the correction parameters (general scaling) observed in the co-occurrence patterns, the default output parameters of this mapping rule are adjusted.

[0051] Adjustment operations are performed using formulas To achieve this.

[0052] in, This is the weight value to be updated, specifically the new default scaling ratio associated with the "apple" morpheme in this example; This is the old weight value before the update, with an initial value of 1.0; It is the learning rate parameter, used to control the step size for each update; It represents the number of corrections to the mapping rule observed within a statistical period; the subscript in the summation symbol. The ordinal number representing the correction operation, traversed from 1 to... ; It is the target parameter value determined by the user after the kth correction, that is, the scaling ratio actually adjusted by the user; It is the initial parameter value generated by the mapping rule before the k-th correction, which is always 1 in this example. .

[0053] Learning rate The setup is based on convergence experiments. In a simulation environment, data was corrected using 10,000 simulated users, and tests were conducted. The convergence rate and stability of the parameters were studied when the values ​​were 0.01, 0.05, 0.1, 0.2, and 0.5. Experimental results show that... In this embodiment, the parameters can stably converge to a relatively optimal value after about 500 iterations, and are not sensitive to noisy data. The value is 0.1.

[0054] Table 2 Example data table of corrected parameters; Table 2 lists some of the corrected data. Example: Assuming the average target scaling factor after 45 corrections is 1.22, that is... Because of all Since both are 1.0, the average deviation is... . ; The result indicates that the new weight values ​​should be adjusted from 1.0 to 1.022. The calculated updated weight values... Replace the original The data is stored in the "apple" entry of the lexical knowledge base under the default size parameters, and an updated lexical knowledge base is generated.

[0055] The above embodiments illustrate preferred embodiments of the present invention. Any equivalent adjustments to the technical solution based on software engineering methods are within the scope of protection, including but not limited to: implementing algorithm logic using different programming languages, refactoring functional modules into services, adjusting data interaction protocols, and optimizing resource scheduling strategies. Any implementation scheme derived from reasonable modifications to the data processing flow, service call chain, or system architecture layer without departing from the core technology of the present invention should be considered within the scope of protection defined by the claims of the present invention.

Claims

1. A digital mural creation system integrating voice control, characterized in that, The system includes: a voice command parsing module, which acquires audio waveform data, separates voice segments from the audio waveform data, parses out text strings carrying timestamps, and transmits the text strings carrying timestamps to the creative intent recognition module; The creative intent recognition module segments the text string, matches word units with the word knowledge base to assign category labels, and judges the category label sequence to generate structured creation instructions, and transmits the structured creation instructions to the graphic element generation module. The graphic element generation module parses the structured creation instructions and generates vector path control points. Then, it combines the brush parameter set to stroke the vector path control points, calculates the canvas element dataset, and passes the canvas element dataset to the instruction mapping self-learning module. The instruction mapping self-learning module monitors the correction instructions of the canvas element dataset, records the initial instructions and correction parameters, analyzes the co-occurrence patterns, and then adjusts the mapping rules of the lexical knowledge base according to the co-occurrence patterns.

2. The mural digital creation system integrating voice control according to claim 1, characterized in that, The text string carrying the timestamp specifically includes a translated word sequence and timestamp information; The structured creation instructions include creation object words, painting action words, attribute description words, spatial composition words, and adjustment and correction words; The canvas element dataset specifically refers to vector path data, brush parameters, and color information; The mapping rules specifically consist of a set of instruction tuples and corresponding operation parameters.

3. The mural digital creation system integrating voice control according to claim 1, characterized in that, The voice command parsing module includes: an audio data acquisition submodule, which acquires the audio waveform data and performs preprocessing to generate the original audio stream; The voice activity detection submodule calculates the short-time energy and zero-crossing rate of each audio frame based on the original audio stream, determines the start and end points of voice activity according to preset energy thresholds and zero-crossing rate thresholds, and separates the voice segments from the original audio stream to obtain independent voice data. The speech content translation submodule calls the acoustic model and language model to recognize the independent speech data and outputs a translated word sequence without time information; The timestamp alignment submodule integrates the start and end points of the speech activity and the translated word sequence, and assigns a start time and duration to each word unit in the translated word sequence through a forced alignment algorithm to generate the text string carrying the timestamp. The preset energy threshold includes a high energy threshold and a low energy threshold. The high energy threshold is used to initially determine the existence of the speech core region, and the low energy threshold is used to define the start and end boundaries of the speech segment. The zero-crossing rate threshold includes a high zero-crossing rate threshold and a low zero-crossing rate threshold, which are used to help distinguish between voiceless and voiced components in speech.

4. The mural digital creation system integrating voice control according to claim 1, characterized in that, The creative intent recognition module includes: a text string segmentation submodule, which obtains the text string carrying the timestamp, and segments it into multiple word units containing independent semantics according to the maximum matching method and a pre-set dictionary, generating a segmentation result set; The word category matching submodule traverses each word unit in the word segmentation result set, queries the word knowledge base to match and assign a category label from the following categories: creative object, painting action, attribute description, spatial composition, or adjustment and correction, and generates a label sequence. The syntax structure analysis submodule analyzes the permutations and combinations of the tag sequences, matches the tag sequences using a predefined authoring instruction syntax tree, identifies the instruction structure pattern, and obtains the instruction template. The creation instruction generation submodule, based on the instruction template, fills the word units carried in the tag sequence into the corresponding parameter slots of the instruction template to generate the structured creation instruction.

5. The mural digital creation system integrating voice control according to claim 1, characterized in that, The graphic element generation module includes: an instruction content parsing submodule, which receives the structured creation instruction and extracts key parameters such as the type of creation object, spatial position, color and brush style from it to generate a standardized parameter list; The control point positioning submodule calls the corresponding geometric primitive model according to the creation object type in the standardized parameter list, and calculates a set of initial vector path control points in the canvas coordinate system in combination with the spatial position to obtain the control point coordinate set. The path curve fitting submodule uses a cubic Bézier curve algorithm to perform interpolation and smoothing on the control point coordinate set, generating a smooth vector path connecting all vector path control points and obtaining vector path data. The brush effect rendering submodule calls the parameters in the brush parameter set that match the brush style in the standardized parameter list to perform outlining and filling rendering on the vector path data, calculates the rendered pixel information, and integrates the vector path data, the selected brush parameters and color information to generate the canvas element dataset.

6. The mural digital creation system integrating voice control according to claim 1, characterized in that, The instruction mapping self-learning module includes: a user correction monitoring submodule, which continuously monitors subsequent manual adjustment operations for the canvas element dataset after the dataset is generated. When an adjustment operation is detected, the adjustment correction terminology and specific correction parameters corresponding to the operation are recorded, and a correction event record is generated. The instruction association analysis submodule traces back to the most recently generated canvas element dataset before the correction event occurred, and indexes the structured creation instruction that generated the dataset, establishing the association between the initial instruction, canvas elements, and correction operation to obtain the instruction correction pair; The co-occurrence pattern mining submodule statistically analyzes multiple stored instruction correction pairs, uses an association rule algorithm to calculate the co-occurrence frequency and confidence between different initial instructions and correction parameters, filters out association patterns that are higher than a preset confidence threshold, and generates the co-occurrence pattern. The knowledge base rule update submodule locates the mapping rule related to the initial instruction in the lexical knowledge base based on the co-occurrence pattern, and adjusts the default output parameter of the mapping rule according to the correction parameter to generate the updated lexical knowledge base.

7. The mural digital creation system integrating voice control according to claim 4, characterized in that, When identifying the instruction structure pattern, the syntax structure analysis submodule specifically performs the following operation: for the tag sequence, it generates multiple candidate instruction structure patterns. According to claim 4, the mural digital creation system integrating voice control is characterized in that, when the grammatical structure analysis submodule identifies the instruction structure pattern, the specific operation is as follows: for the tag sequence, generate multiple candidate instruction structure patterns; The instruction confidence score for each candidate instruction structure pattern is calculated using the following formula. : ； in, The instruction confidence level represents the candidate instruction structure pattern. This represents the word unit at the i-th position in the label sequence. The representative assigns the term unit Category tags, This represents the word unit recorded in the lexical knowledge base. Belongs to the category label The prior probability, Represents all category labels The complete category label sequence constitutes This represents the structural probability that the complete category label sequence conforms to predefined grammar rules. Represents the weighting coefficient. The counting variable is the summation symbol, and its range is from 1 to n. The length of the label sequence; Finally, the command confidence level is selected. The highest-ranking candidate instruction structure pattern is selected as the instruction structure pattern, and the instruction template is generated.

8. The mural digital creation system integrating voice control according to claim 5, characterized in that, When the control point positioning submodule calculates the initial vector path control points, if the standardized parameter list contains spatial composition terms, its specific operation is as follows: obtain the canvas bounding box coordinates of the reference object specified in the structured creation instruction; Based on the semantics of the spatial composition lexical units, a target generation region is calculated based on the canvas bounding box coordinates of the reference object; Invoke the geometric primitive model corresponding to the type of the creation object. This model pre-determines the number and relative position distribution of control points required to describe the outline of the object. The relative position distribution of the model is linearly mapped to the target generation area, and adjusted in combination with random offsets. The absolute coordinates of each control point in the canvas coordinate system are calculated to generate the control point coordinate set. The absolute coordinates of each control point in the canvas coordinate system are calculated using the following formula: ; in, This represents the final absolute coordinates of the j-th control point. This represents the coordinates of the upper left corner reference point of the target generation area. This represents the width of the target generation region. This represents the height of the target generation region. This represents the normalized relative position coordinates of the j-th control point obtained from the geometric primitive model, and its value ranges from 0 to 1. The perturbation coefficient represents the control of the random offset amplitude. Represents a standard normally distributed random number. The index used to identify control points.

9. The mural digital creation system integrating voice control according to claim 6, characterized in that, When the knowledge base rule update submodule adjusts the mapping rule, the specific operation is as follows: locate the specific operation parameter in the mapping rule that directly corresponds to the correction parameter; The weight values ​​of the specific operation parameters are updated using the following formula. : ; in, This represents the updated weight value. This represents the weight value of the specific operation parameter before the update. Represents the learning rate parameter. This represents the number of corrections to the mapping rule observed within a statistical period. This represents the target parameter value determined by the user after the k-th correction. This represents the initial parameter value generated by the mapping rule before the k-th correction. The counting variable is for the summation sign, and its range is from 1 to N; The calculated updated weight values Replace the original The updated lexical knowledge base is generated.

10. The mural digital creation system integrating voice control according to claim 3, characterized in that, When the voice activity detection submodule separates the voice segments, the specific operation is as follows: the acquired original audio stream is divided into continuous audio frames of fixed duration; For each audio frame, calculate its short-time energy and short-time average zero-crossing rate, and generate an energy parameter sequence and a zero-crossing rate parameter sequence. By setting high-energy threshold, low-energy threshold, high zero-crossing rate threshold and low zero-crossing rate threshold, the start and end points of speech can be preliminarily identified by comparing the energy parameter sequence with the zero-crossing rate parameter sequence. A dual-threshold algorithm is used to verify the determined speech start and end points. Specifically, the speech core area determined by the high threshold is extended to both sides until it is below the low threshold to determine the boundary of the speech segment. Waveform data within the boundary is then extracted from the original audio stream to obtain the independent speech data.

Images