A multi-modal AI-based audio intelligent processing method and system for sound comic

Abstract

The application discloses a kind of based on multimodal AI's audio intelligent processing method and system of sound comic, it is related to audio processing technical field.The application obtains comic split shot image and text script data, carries out denoising enhancement to split shot image, and identifies text area extraction visual position information;Respectively carry out feature coding to preprocessed image and text script, complete cross-modal feature fusion in conjunction with visual position information, generate three-dimensional emotional feature vector containing emotional intensity, spatial position and semantic priority according to comic reading time sequence;Rely on feature vector to determine speech synthesis parameter and scene sound effect matching strategy, respectively generate character voice and scene sound effect data;Again, quality assessment and synchronous calibration are carried out to two kinds of audio data, according to comic split shot reading time sequence splicing, generate target audio file with picture accurate synchronization.The application realizes comic image-text multimodal intelligent analysis and audio automatic adaptation generation, improves sound comic production efficiency and audio-visual synchronization effect.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to a method and system for intelligent processing of audio in audio comics based on multimodal AI. Background Technology

[0002] With the growing demand for digital reading and fragmented entertainment, audio comics, as an emerging medium integrating visual and auditory elements, have a core requirement of achieving an immersive narrative effect through the precise matching of audio and visual content. This includes emotional adaptation of character voices, atmospheric creation of scene sound effects, and collaborative processing of multimodal data. However, the industry currently faces significant technical bottlenecks in audio processing, hindering content production efficiency and user experience improvement.

[0003] (a) Limitations of single-mode technology Traditional audio and visual processing technologies have long relied on a single information source, limiting processing accuracy and application scenarios. For example, in speech synthesis, while traditional TTS systems have achieved basic speech generation, they generally lack the ability to dynamically adapt to emotions. They largely depend on preset speech templates and cannot adjust tone, speed, and timbre in real time based on the facial expressions of comic book characters (such as anger, joy, and sadness) or the emotional tone of dialogue (such as rhetorical questions, exclamations, and statements). The accuracy rate of emotion recognition is only 58.3%, far lower than the 91.7% level of human voice actors. In image understanding, CNN-based static image analysis technology cannot effectively capture the dynamic narrative logic between comic book panels, resulting in an automatic panel recognition error rate as high as 23.5%, directly affecting the synchronization accuracy of audio and visual content.

[0004] (ii) Lack of multimodal data collaboration mechanism Audio comic creation involves three core data types: images, text, and audio. Deep collaboration between these three types is crucial for achieving a high-quality experience. Current technology has failed to effectively establish cross-modal relationships: on the one hand, scene features in comic images (such as rainy nights or battlefields) and character postures (such as body movements and facial expressions) cannot directly guide sound effect generation; on the other hand, the mapping relationship between the semantic emotion of text dialogue and audio parameters (such as volume, frequency, and pauses) has not yet formed a standardized model. Typical problems include a disconnect between scene sound effects and visual atmosphere (such as tense battle scenes paired with calm background music) and synchronization errors between character voice and lip movements, severely affecting narrative coherence.

[0005] (III) The pain points of efficiency and cost in industry creation Traditional audio comic production heavily relies on professional voice actors and post-production teams, resulting in high costs and lengthy production cycles. Data shows that manually producing a single episode (approximately 20 minutes) of audio content takes an average of 12 hours, including multiple stages such as script analysis, comic panel drawing, character voice acting, and sound effect synthesis, requiring repeated revisions to match the visual pacing. This model not only struggles to meet the demands of large-scale content production but also suffers from uneven distribution of human resources and high costs associated with regional collaboration, leading to limited content production capacity and insufficient personalized expression within the industry. Summary of the Invention

[0006] The present invention aims to at least partially solve one of the technical problems in the related art.

[0007] Therefore, the first objective of this invention is to propose a method for intelligent audio processing of audio comics based on multimodal AI, comprising the following steps: S1, acquire the comic panel image to be processed and the corresponding text dialogue data, perform noise reduction and enhancement processing on the comic panel image to obtain a preprocessed image, and identify the text region in the preprocessed image to extract visual position information. S2, respectively encode the preprocessed image and the text dialogue data to obtain image feature vector and text feature vector, use the visual position information to perform cross-modal fusion of the text feature vector and the image feature vector, and combine the reading time sequence of the comic panel to generate a three-dimensional emotional feature vector containing emotional intensity, spatial position and semantic priority. S3. Based on the three-dimensional emotion feature vector, determine the speech synthesis parameters and scene sound effect matching strategy, use the pre-trained speech generation model in combination with the speech synthesis parameters to generate character speech data, and retrieve scene sound effect data from the sound effect library or synthesize it through the generation model according to the scene sound effect matching strategy. S4, perform quality assessment and synchronization calibration on the character voice data and the scene sound effect data, and splice the calibrated audio data according to the reading sequence of the comic panel to generate a target audio file synchronized with the comic panel screen.

[0008] In one embodiment of the present invention, S1 includes: S11, a bilateral filtering algorithm is used to perform pixel-level denoising on the comic panel image to eliminate high-frequency noise interference, and the multi-scale decomposition technique based on Retinex theory is used to perform contrast enhancement processing on the denoised image to obtain an enhanced image with clear details. S12, the text detection network combining CTPN and Mask R-CNN models is used to scan and locate the dialog box region in the enhanced image, extract visual position information containing the normalized coordinates of the center point of the text box, and convert the visual position information into an embedding vector for subsequent feature fusion.

[0009] In one embodiment of the present invention, S12 includes: S121, perform text line detection on the enhanced image using the CTPN model to generate a candidate text region sequence, and input the candidate text region sequence into the Mask R-CNN model for instance segmentation to accurately define the dialog box boundary; S122, calculate the normalized coordinates of the center point of the character's facial region based on the dialog box boundary, and map the normalized coordinates into a two-dimensional spatial position vector, which is then output to the cross-modal fusion stage as a core component of the visual position information.

[0010] In one embodiment of the present invention, S2 includes: S21, feature extraction is performed on the preprocessed image based on the improved CLIP model. The stylized elements unique to comics are captured through the newly added halftone texture feature extraction branch, generating an image feature vector containing global semantics and local texture. S22, The text dialogue data is encoded using the BERT-Base model that incorporates visual position encoding, and the visual position information is converted into an embedding vector and superimposed on the text semantic features to generate a text feature vector; S23, construct a multi-head attention mechanism, calculate the association weight between the text feature vector as the query vector and the image feature vector as the key vector and value vector, and generate multi-head attention output features; S24, the multi-head attention output features are input into the gated recurrent unit (GRU), and temporal encoding is performed according to the reading order of the comic panel. The hidden state containing temporal context information is output, and the hidden state is concatenated with the multi-head attention output features and then passed through a linear projection layer to generate a cross-modal feature vector. S25, Based on the cross-modal feature vector, construct a three-dimensional emotion feature vector: V_emotion=[EI, SP_x, SP_y, Priority], Among them, the emotional intensity EI is obtained by predicting the amplitude of facial muscle movements and the dot density through CNN regression, the spatial positions SP_x and SP_y are derived from the visual position information, and the semantic priority is obtained by normalizing the emotional saliency calculated by the attention mechanism.

[0011] In one embodiment of the present invention, S23 includes: S231, the text feature vector is multiplied by the query weight matrix of the i-th attention head to obtain the query vector, and the image feature vector is multiplied by the key weight matrix and value weight matrix of the i-th attention head to obtain the key vector and value vector respectively. S232, calculate the output of the i-th attention head according to the single-head attention weight formula, wherein the single-head attention weight formula is:

[0012] Where, d k It is a single-head dimension; S233, the parallel computation process is executed to learn local facial expressions, global scene, text tone and image focus features respectively, the outputs of all attention heads are concatenated and linearly projected through the output weight matrix to obtain the multi-head attention output features.

[0013] In one embodiment of the present invention, S24 includes: S241, the attention features of the scene at time t and the hidden state of the previous time step are input into the update gate and the reset gate, respectively. The update gate control signal is calculated according to the update gate formula, and the reset gate control signal is calculated according to the reset gate formula. The update gate formula is as follows: ; The reset gate formula is: ; S242, the hidden state of the previous time step is weighted and filtered using the reset gating signal, and candidate hidden states are calculated by combining the attention features of the current time step: ; S243, based on the updated gating signal, linear interpolation is performed to fuse the hidden state of the previous time step with the candidate hidden state, and the temporal output hidden state at time t is generated according to the output hidden state formula to represent the dynamic narrative logic between comic panel scenes. The output hidden state formula is: .

[0014] In one embodiment of the present invention, S3 includes: S31, analyze the emotional intensity EI and semantic priority in the three-dimensional emotional feature vector, dynamically adjust the speech rate fluctuation range and pitch change amplitude in the speech synthesis parameters, and set the spatial reverberation parameters that are dynamically adjusted based on the spatial location and the early reflection coefficient. S32, adopting a GAN-enhanced end-to-end TTS architecture, injects the speech synthesis parameters into the style embedding layer of the decoder, extracts the speaker style features of the reference audio through the StyleGAN2 network and generates style vectors to control timbre and prosody, and generates the intermediate Mel spectrum; S33, the intermediate Mel spectrum is input to the MelGAN vocoder to perform waveform synthesis, and the output is character voice data with specific emotional expression and spatial positioning effect; S34, the Mask R-CNN segmentation network is used to identify the visual semantic parameters in the preprocessed image to determine the scene classification label, and a three-dimensional emotion space model including valence, arousal and dominance is constructed based on the three-dimensional emotion feature vector. S35, calculate the cosine similarity between the feature vector formed by the scene classification label and the three-dimensional emotional space model and the center vector of each category in the sound effect knowledge base. When the highest similarity is greater than or equal to the set threshold, retrieve the scene sound effect data with the highest matching degree from the sound effect sample knowledge base. S36. When the highest similarity is less than the set threshold and is determined to be a neutral or non-covered scene in the knowledge base, the Diffusion Model is adopted with the visual semantic parameters and the three-dimensional emotion space model as conditional inputs.

[0015] In one embodiment of the present invention, S4 includes: S41, perform first-level objective index detection to verify whether the speech clarity PESQ is greater than or equal to the clarity threshold, whether the emotion matching degree is greater than or equal to the matching threshold, and whether the background noise is less than or equal to the noise threshold, and perform second-level automated quality scoring through a pre-trained audio quality assessment model based on CNN-LSTM architecture; S42, Generate a JSON-formatted synchronization information file containing the mapping relationship between voice segments and comic panel timestamps based on the reading sequence of the comic panel, and use a dynamic time warping algorithm to align the character voice data and the scene sound effect data at the millisecond level; S43 seamlessly splices the aligned audio data according to the scene sequence and exports it as the target audio file.

[0016] To achieve the above objectives, a second aspect of the present invention proposes an intelligent audio processing system for audio comics based on multimodal AI, comprising the following steps: The image and text preprocessing module is used to acquire the comic panel image to be processed and the corresponding text dialogue data, perform noise reduction and enhancement processing on the comic panel image to obtain a preprocessed image, and identify the text region in the preprocessed image to extract visual position information. The emotional feature fusion module is used to encode the preprocessed image and the text dialogue data to obtain image feature vectors and text feature vectors respectively, use the visual position information to perform cross-modal fusion of the text feature vectors and the image feature vectors, and combine the reading time sequence of the comic panel to generate a three-dimensional emotional feature vector containing emotional intensity, spatial position and semantic priority. The speech and sound effect generation module is used to determine the speech synthesis parameters and scene sound effect matching strategy based on the three-dimensional emotion feature vector, generate character speech data by combining the pre-trained speech generation model with the speech synthesis parameters, and retrieve scene sound effect data from the sound effect library or synthesize it through the generation model according to the scene sound effect matching strategy. The audio timing synthesis module is used to perform quality assessment and synchronization calibration on the character voice data and the scene sound effect data, and to splice the calibrated audio data according to the reading sequence of the comic panel to generate a target audio file synchronized with the comic panel screen.

[0017] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0018] The methods, systems, and storage media of this invention have the following beneficial effects: 1. More precise expression of emotions Multimodal fusion technology, implemented through an emotion matching module, deeply integrates image visual features with text semantic information to generate fine-grained emotion tag sequences. Experimental data shows that this technology increases the accuracy of emotion matching from 60% to over 90% compared to existing technologies. It can accurately capture the emotional changes of comic book characters in different scenarios, such as the subtle expression of complex emotions like anger, joy, and sadness, significantly enhancing the emotional impact of audio content.

[0019] 2. Processing efficiency breaks through industry bottlenecks Employing a distributed parallel processing architecture, the system achieves a storyboard processing speed of 3 frames per second, three times faster than the industry average; the complete audio generation cycle is reduced to 1 / 8 of the traditional manual production process. Taking a 200-page comic as an example, the system's end-to-end processing time is only 45 minutes, while a traditional studio would need 6-8 hours to complete the same workload.

[0020] 3. Costs are significantly reduced. Through the synergistic effect of the intelligent speech synthesis module and the automated sound effect matching module, 30%-50% of the manual dubbing work is replaced, reducing the production cost of a single work by 42%; the intelligent optimization module reduces the audio post-processing time by 60%, saving about 35% in labor costs. Calculated for a medium-sized studio producing 100 works per year, the annual cost savings can reach 1.2-1.5 million yuan.

[0021] 4. Highly personalized and adaptable The system features a built-in custom resource management module, allowing users to upload over 50 basic sound effects to build personalized character sound libraries, and provides a sound effect library covering over 80 scene types. Whether it's action-packed battles, slice-of-life school life, or suspenseful mystery comics, the flexible resource configuration can meet diverse creative needs, enhancing the product's market adaptability.

[0022] 5. Multimodal collaborative optimization Through the precise timing calibration algorithm of the audio-visual synchronization control module, the synchronization rate of audio, visuals, and text is improved to 98%. This module analyzes the rhythm of comic panel transitions and the duration of text dialogues, dynamically adjusting the speech synthesis speed and the timing of sound effect insertion, fundamentally solving the "audio-visual disconnect" problem commonly found in traditional technologies, and improving the user's auditory and visual collaborative experience.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of an audio intelligent processing method for audio comics based on multimodal AI according to an embodiment of the present invention; Figure 2 This is a structural diagram of an audio intelligent processing system for audio comics based on multimodal AI according to an embodiment of the present invention; Figure 3 This is a system structure framework diagram according to an embodiment of the present invention; Figure 4 This is a flowchart of a processing method according to an embodiment of the present invention. Detailed Implementation

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] The following description, with reference to the accompanying drawings, describes an embodiment of the present invention of an intelligent audio processing method and system for audio comics based on multimodal AI.

[0028] Example 1 Figure 1 This is a flowchart of an embodiment of the present invention for a method of generating audio for audio comics based on multimodal data.

[0029] like Figure 1 As shown, the method for generating audio for audio comics based on multimodal data includes the following steps: S1. Obtain the comic panel image to be processed and the corresponding text dialogue data. Perform denoising and enhancement processing on the comic panel image to obtain a preprocessed image. Identify the text region in the preprocessed image to extract visual position information.

[0030] This step aims to address the technical problem of insufficient feature alignment accuracy in the multimodal data input stage due to image noise interference and missing text spatial information. Its core lies in constructing a standardized data preprocessing mechanism for comic panel storyboarding, establishing an initial multimodal input foundation by simultaneously acquiring visual image data and associated text dialogue data.

[0031] In the processing, denoising and enhancement operations are first performed on the original comic storyboard images to eliminate visual artifacts introduced by acquisition or compression and improve the recognizability of key features, thereby obtaining high-quality preprocessed images. Subsequently, based on the preprocessed images, the text regions carrying dialogue content are located and segmented, and then the visual position information representing the distribution pattern of these regions in the picture is analyzed and extracted. This visual position information not only includes the geometric coordinates of the text, but also implies its spatial semantics in the storyboard narrative logic, providing necessary spatial anchors for subsequent cross-modal fusion.

[0032] Specifically, step S1 includes: S11, a bilateral filtering algorithm is used to perform pixel-level denoising on the comic panel image to eliminate high-frequency noise interference, and the multi-scale decomposition technique based on Retinex theory is used to perform contrast enhancement processing on the denoised image to obtain an enhanced image with clear details. S12, the text detection network combining CTPN and Mask R-CNN models is used to scan and locate the dialog box area in the enhanced image, extract visual position information containing the normalized coordinates of the center point of the text box, and convert the visual position information into an embedding vector for subsequent feature fusion to achieve adaptive processing of images with different resolutions from 720P to 4K.

[0033] S12 includes: S121, perform text line detection on the enhanced image using the CTPN model to generate a candidate text region sequence, and input the candidate text region sequence into the Mask R-CNN model for instance segmentation to accurately define the dialog box boundary; S122, calculate the normalized coordinates of the center point of the character's facial region based on the dialog box boundary, and map the normalized coordinates into a two-dimensional spatial position vector, which is then output to the cross-modal fusion stage as a core component of the visual position information.

[0034] By performing this step, the input standards of multi-source heterogeneous data can be effectively unified, the interference of noise on feature extraction can be significantly reduced, and the spatial correspondence between text and images can be accurately captured. This lays a solid data foundation for the subsequent construction of high-precision cross-modal emotion feature vectors, thereby improving the synchronization and emotional adaptability of the overall audio generation.

[0035] S2, feature encoding is performed on the preprocessed image and the text dialogue data respectively to obtain image feature vector and text feature vector. The text feature vector and the image feature vector are fused across modally using the visual position information, and a three-dimensional emotional feature vector containing emotional intensity, spatial position and semantic priority is generated by combining the reading time sequence of comic panel.

[0036] This step aims to extract deep semantic features from preprocessed image and text dialogue data using a dual-path parallel encoding architecture, thereby constructing a unified cross-modal representation space. Specifically, the system first uses an improved visual encoding model to capture global scene information and unique stylized textures in comic panel layouts. Simultaneously, it employs a text encoding model incorporating visual position encoding to semantically parse the dialogue data. Visual position encoding transforms the coordinate information of text regions in the image into embedding vectors to enhance spatial relevance. Subsequently, based on a hybrid fusion strategy combining self-attention mechanisms and temporal recursive units, the system calculates the dynamic correlation weights between image region features and text fragment features, and performs temporal modeling of the fused features according to the reading order of the comic panel layouts. During this process, the system comprehensively regresses and predicts emotional intensity using visual cues such as facial muscle movement amplitude, limb intensity, and dot density. It then determines the spatial location by combining the normalized coordinates of the character's facial region center point and calculates the semantic priority through an attention mechanism. Finally, it generates a three-dimensional emotional feature vector containing emotional intensity, spatial location, and semantic priority.

[0037] Specifically, step S2 includes: S21, feature extraction is performed on the preprocessed image based on the improved CLIP model. The newly added halftone texture feature extraction branch captures the stylized elements unique to comics and generates an 896-dimensional image feature vector containing global semantics and local texture.

[0038] S22, the text dialogue data is encoded using the BERT-Base model that incorporates visual position encoding, the visual position information is converted into an embedding vector and superimposed on the text semantic features to generate a 768-dimensional text feature vector.

[0039] S23, construct a multi-head attention mechanism containing 16 attention heads, calculate the association weight between the text feature vector as the query vector and the image feature vector as the key vector and value vector, and generate multi-head attention output features.

[0040] S23 includes: S231, the text feature vector is multiplied by the query weight matrix of the i-th attention head to obtain the query vector, and the image feature vector is multiplied by the key weight matrix and value weight matrix of the i-th attention head to obtain the key vector and value vector respectively. S232, calculate the output of the i-th attention head according to the single-head attention weight formula, wherein the single-head attention weight formula is:

[0041] Where, d k It is a single-head dimension; S233, the parallel computation process is executed to learn local facial expressions, global scene, text tone and image focus features respectively, the outputs of all attention heads are concatenated and linearly projected through the output weight matrix to obtain the multi-head attention output features.

[0042] S24, the multi-head attention output features are input into the gated recurrent unit (GRU), and temporal encoding is performed according to the reading order of the comic panel. The hidden state containing temporal context information is output, and the hidden state is concatenated with the multi-head attention output features and passed through a linear projection layer to generate a cross-modal feature vector.

[0043] S24 includes: S241, the attention features of the scene at time t and the hidden state of the previous time step are input into the update gate and the reset gate, respectively. The update gate control signal is calculated according to the update gate formula, and the reset gate control signal is calculated according to the reset gate formula. The update gate formula is as follows: ; The reset gate formula is: ; S242, the hidden state of the previous time step is weighted and filtered using the reset gating signal, and candidate hidden states are calculated by combining the attention features of the current time step: ; S243, based on the updated gating signal, linear interpolation is performed to fuse the hidden state of the previous time step with the candidate hidden state, and the temporal output hidden state at time t is generated according to the output hidden state formula to represent the dynamic narrative logic between comic panel scenes. The output hidden state formula is: .

[0044] S25, Based on the cross-modal feature vector, construct a three-dimensional emotion feature vector: V_emotion=[EI, SP_x, SP_y, Priority], Among them, the emotional intensity EI is obtained by predicting the amplitude of facial muscle movements and the dot density through CNN regression, the spatial positions SP_x and SP_y are derived from the visual position information, and the semantic priority is obtained by normalizing the emotional saliency calculated by the attention mechanism.

[0045] By constructing this deeply integrated cross-modal feature representation mechanism, this step effectively solves the technical challenge of the separation between visual symbols and textual semantics in traditional single-modal processing, significantly improving the accuracy and fine-grainedness of emotion recognition in complex comic scenes. Simultaneously, by introducing the storyboard reading sequence and a three-dimensional emotion dimension, the generated feature vectors can accurately map character emotional changes and spatial narrative logic, providing a high-confidence data foundation for subsequent adjustments to voice emotion parameters and adaptive matching of scene sound effects, thereby significantly reducing the emotion mismatch rate of multimodal content.

[0046] S3. Based on the three-dimensional emotion feature vector, determine the speech synthesis parameters and scene sound effect matching strategy. Use a pre-trained speech generation model combined with the speech synthesis parameters to generate character speech data. Then, retrieve scene sound effect data from the sound effect library or synthesize it through the generation model according to the scene sound effect matching strategy. The process of determining speech synthesis parameters and scene sound effect matching strategies based on three-dimensional emotion feature vectors, thereby driving audio data generation, hinges on establishing a mapping mechanism from multimodal emotion representations to acoustic physical quantities. This step first analyzes the emotion intensity, spatial location, and semantic priority information contained in the three-dimensional emotion feature vectors, transforming them into synthesis parameters that control speech prosody, timbre style, and sound field localization. Simultaneously, sound effect matching rules are derived based on scene semantics. Subsequently, a pre-trained speech generation model is invoked, fusing the text content with the aforementioned synthesis parameters. The encoder extracts semantic features, and the decoder combines these with style vectors to generate character speech data with specific emotional nuances. Regarding scene sound effect processing, matching is performed in a structured sound effect library based on the matching strategy. Alternatively, when resources in the library cannot cover all elements, a generative model is used to synthesize appropriate background sound effect data in real time, conditioned on scene features.

[0047] Specifically, step S3 includes: S31, analyze the emotional intensity EI and semantic priority in the three-dimensional emotional feature vector, dynamically adjust the speech rate fluctuation range and pitch change amplitude in the speech synthesis parameters, and set the spatial reverberation parameters for dynamic adjustment of the early reflection coefficient according to the spatial location.

[0048] In one specific implementation, the speech rate fluctuation range is ±15%, the pitch change range is ±200Hz, and the early reflection coefficient is dynamically adjusted between 0.3 and 0.7.

[0049] S32 employs a GAN-enhanced end-to-end TTS architecture, injecting the speech synthesis parameters into the style embedding layer of the decoder. The StyleGAN2 network extracts 512-dimensional speaker style features from the reference audio and generates style vectors to control timbre and prosody, generating an intermediate Mel spectrum.

[0050] S33, the intermediate Mel spectrum is input to the MelGAN vocoder to perform waveform synthesis, and the output is character voice data with specific emotional expression and spatial positioning effect.

[0051] S34, the Mask R-CNN segmentation network is used to identify 12 types of visual semantic parameters in the preprocessed image to determine scene classification labels, and a three-dimensional emotion space model including valence, arousal and dominance is constructed based on the three-dimensional emotion feature vector.

[0052] S35, calculate the cosine similarity between the feature vector formed by the scene classification label and the three-dimensional emotional space model and the center vector of each category in the sound effect knowledge base. When the highest similarity is greater than or equal to a set threshold, retrieve the scene sound effect data with the highest matching degree from the knowledge base containing more than 5,000 sound effect samples.

[0053] As a specific implementation method, the threshold is set to 0.6.

[0054] S36. When the highest similarity is less than the set threshold and is determined to be a neutral or non-covered scene in the knowledge base, the Diffusion Model is adopted with the visual semantic parameters and the three-dimensional emotion space model as conditional inputs.

[0055] This step, through the refined guidance of three-dimensional emotional feature vectors in the audio generation process, achieves a high degree of unity between visual semantics and auditory expression, effectively solving the problems of singular emotional expression and audio-visual disconnect in traditional technologies. Utilizing a hybrid sound effect processing strategy that combines retrieval and generation, it ensures the matching accuracy of sound effects for common scenes while also possessing the ability to generalize generation capabilities for long-tailed and complex scenes, significantly improving the immersive narrative effect and automated production level of audio comics.

[0056] S4, perform quality assessment and synchronization calibration on the character voice data and the scene sound effect data, and splice the calibrated audio data according to the reading sequence of the comic panel to generate a target audio file synchronized with the comic panel images. This step aims to address the technical challenges of inconsistent quality among components and audio-visual timing misalignment during multimodal audio generation. Its core lies in constructing a post-processing mechanism that integrates quality quantification assessment, multi-source data synchronous calibration, and temporal splicing. Specifically, this mechanism first performs multi-dimensional quality assessments on independently generated character voice data and scene sound effect data. Through the synergy of objective indicator detection and a pre-trained evaluation model, it determines the clarity, emotional matching, and background noise level of the audio signal, ensuring that each component meets preset quality thresholds. Subsequently, based on the reading sequence logic of comic panel layouts, it synchronously calibrates the assessed audio data, dynamically adjusting the start time and duration of voice and sound effect segments using timestamp mapping relationships to eliminate audio-visual asynchrony caused by generation delays or rhythm differences. Finally, it seamlessly splices the calibrated multiple audio segments according to the narrative order of the panel layouts to generate a target audio file that is strictly synchronized with the comic panel visuals.

[0057] Step S4 includes: S41, perform first-level objective index detection to verify whether the speech clarity PESQ is greater than or equal to the clarity threshold, whether the emotion matching degree is greater than or equal to the matching threshold, and whether the background noise is less than or equal to the noise threshold, and perform second-level automated quality scoring through a pre-trained audio quality assessment model based on the CNN-LSTM architecture.

[0058] As one specific implementation, the sharpness threshold is 3.8, the matching threshold is 85%, and the noise threshold is -60dB.

[0059] S42, generate a JSON-formatted synchronization information file containing the mapping relationship between voice segments and comic panel timestamps based on the reading sequence of the comic panel, and use a dynamic time warping algorithm to align the character voice data and the scene sound effect data at the millisecond level to ensure that the audio-visual synchronization error is less than or equal to 50ms.

[0060] S43 seamlessly splices the aligned audio data according to the scene sequence and exports it as the target audio file.

[0061] As one specific implementation method, the target audio file can be in MP3, WAV, or AAC format.

[0062] Through the aforementioned quality assessment and synchronous calibration mechanism, low-quality audio components can be effectively eliminated and timing deviations can be corrected, significantly improving the auditory consistency and narrative coherence of the final output audio. This ensures that the audio content and comic panel layout are highly consistent in terms of emotional expression and temporal rhythm, thereby achieving an immersive multimodal audiovisual experience.

[0063] Example 2 like Figure 2As shown, an audio intelligent processing system 10 for audio comics based on multimodal AI includes the following steps: The image and text preprocessing module 100 is used to acquire the comic panel image to be processed and the corresponding text dialogue data, perform noise reduction and enhancement processing on the comic panel image to obtain a preprocessed image, and identify the text region in the preprocessed image to extract visual position information. The emotional feature fusion module 200 is used to encode the preprocessed image and the text dialogue data respectively to obtain image feature vectors and text feature vectors, use the visual position information to perform cross-modal fusion of the text feature vectors and the image feature vectors, and combine the reading time sequence of the comic panel to generate a three-dimensional emotional feature vector containing emotional intensity, spatial position and semantic priority. The speech and sound effect generation module 300 is used to determine the speech synthesis parameters and scene sound effect matching strategy based on the three-dimensional emotion feature vector, generate character speech data by combining the speech synthesis parameters with a pre-trained speech generation model, and retrieve scene sound effect data from the sound effect library or synthesize it through the generation model according to the scene sound effect matching strategy. The audio timing synthesis module 400 is used to perform quality assessment and synchronization calibration on the character voice data and the scene sound effect data, and to splice the calibrated audio data according to the reading sequence of the comic panel to generate a target audio file synchronized with the comic panel screen.

[0064] Example 3 To further illustrate the system proposed in this invention, this embodiment is described based on specific implementation methods.

[0065] like Figure 4 As shown, this invention constructs a four-order architecture of "data input-feature fusion-intelligent processing-output optimization" and uses the Transformer cross-modal attention mechanism to achieve deep fusion of text / image / audio features and establish a unified feature space.

[0066] The core modules of this system include a multimodal input module, a cross-modal feature fusion module, an intelligent audio generation module, and a quality optimization and output module.

[0067] Among them, such as Figure 3 As shown, the multimodal input module supports the synchronous input of high-resolution images and multi-format audio. The image acquisition unit adopts an adaptive resolution adjustment mechanism, which supports dynamic adaptation from 720P to 4K. The core processing resolution is no less than 1080P, and the preprocessing time of a single comic panel image is controlled within 0.3 seconds.

[0068] The image preprocessing workflow includes denoising (using a bilateral filtering algorithm), contrast enhancement (based on Retinex theory for multi-scale decomposition), and text region detection (combining CTPN and Mask R-CNN models, achieving a text detection accuracy of 98.7%).

[0069] The audio input unit supports mainstream audio formats such as 44.1kHz / 16bit PCM, MP3, and WAV. It has a built-in Automatic Gain Control (AGC) module, which can uniformly adjust the loudness of the input audio to the standard range of -23 LUFS±1.5. The sampling rate conversion uses the SRC algorithm to ensure that the sound quality loss is less than 0.5%.

[0070] Cross-modal feature fusion module This module is the core semantic understanding hub of the system, employing a dual-path feature extraction + dynamic fusion architecture. Dual-path feature extraction is the core architecture of the cross-modal feature fusion module, meaning that the image feature extraction path and the text feature extraction path are encoded independently and in parallel without interfering with each other, before entering the unified fusion layer.

[0071] The image feature extraction path is based on the improved CLIP model (ViT-L / 14 architecture). While retaining the original model's 86.4% image classification accuracy, it adds a branch for extracting halftone texture features unique to comics. The input layer receives the CLIP intermediate feature map (14×14×768); the first layer is a 3×3 convolution + BN + ReLU (outputting 128 channels); the second layer is a 3×3 convolution + BN + ReLU (outputting 256 channels); the third layer is a Gabor filter bank (8 directions × 4 scales = 32 filters); the fourth layer is a 1×1 convolution to fuse features (outputting 128 channels); global average pooling outputs a 128-dimensional vector, which is concatenated with the CLIP's 768-dimensional features to form an 896-dimensional image feature vector, achieving accurate capture of comic stylized elements.

[0072] The text feature extraction path adopts the BERT-Base model (12-layer Transformer), and is specifically optimized for cartoon dialog box text. By introducing visual position encoding (converting the coordinate information of the text box in the image into an embedding vector), the accuracy of text semantic understanding is improved by 12.3%.

[0073] The feature fusion stage adopts a hybrid architecture of attention mechanism and gated recurrent unit (GRU). The self-attention module is responsible for calculating the association weight between image regions and text segments (the number of attention heads is set to 16), while the GRU unit dynamically encodes the temporal fusion features. The final output is a cross-modal feature vector with a dimension of 768. A single fusion process takes about 0.8 seconds.

[0074] Feature fusion numerical representation: Feature fusion includes image features Text features ,in, =48.

[0075] The formula for single-head attention weights is:

[0076] Where Q represents a text query. K is the image key. V is the image value. .

[0077] The expression for 16-head parallel computation is: ;

[0078] Sixteen heads were trained to learn local expressions, global scenes, text tone, and screen focus, and finally spliced ​​together with linear projection.

[0079] Final fusion output

[0080] GRU timing meaning + state update formula + variable description: GRU timing meaning This refers to the reading order of comic panel layouts (panel 1 → panel 2 → panel 3, etc.).

[0081] The update formula for the gated recurrent unit (GRU) includes: Update gate formula: ; Reset door formula: ; Candidate hidden state formula: ; Output hidden state formula: .

[0082] Variable descriptions: xt represents the attention feature of the t-segment; ht represents the temporal output; zt represents the update gate; rt represents the reset gate.

[0083] 768-dimensional output vector: Using a stitched linear projection, maintaining 768 dimensions, the calculation formula is as follows:

[0084] in, For multi-head attention output (768 dimensions). For GRU timing output (768 dimensions) Intelligent audio generation module This module comprises two main subsystems: speech synthesis and sound effect generation.

[0085] The speech synthesis subsystem adopts a GAN-enhanced end-to-end TTS architecture. The basic model is Tacotron 2, with 80-dimensional mel input parameters, a bidirectional LSTM encoder (512x2 layers), location-sensitive attention (256-dimensional context), an autoregressive LSTM decoder (512x2 layers), and a linear output layer + post-net (5-layer CNN). The vocoder uses MelGAN, and the speech naturalness is optimized by introducing the StyleGAN2 network. Specifically, it extracts the speaker style features (512-dimensional) from the reference audio, generates style vectors through a mapping network, and injects them into the Tacotron 2 decoder to control timbre and prosody, improving the MOS score from 4.2 to 4.6 (out of 5). The system has a built-in library of 20 basic timbres and supports personalized speech customization through feature fine-tuning (fine-tuning data can achieve 85% similarity with as little as 5 minutes). The speech synthesis speed is 3.2 times faster than real-time (i.e., generating 10 seconds of speech takes only 3.1 seconds). Fine-tuning method: (1) Extract 5 minutes of target speech samples and remove silence using VAD; (2) Freeze the encoder and post-net, and only fine-tune the style embedding layer (512-dimensional) and attention layer of the decoder; (3) Learning rate 1e-5, Adam optimizer; (4) Fine-tune training for 3-5 epochs, and stop when the mel spectrum similarity reaches 85%.

[0086] The sound effect generation subsystem is based on a hybrid strategy of knowledge graph and generation model, constructing a knowledge base containing 5000+ sound effect samples. It achieves accurate sound effect recommendation through a dual matching mechanism (three-dimensional emotional space model - (1) Valence [-1,1]: positive / negative degree; (2) Arousal [0,1]: intensity; (3) Dominance [0,1]: control degree. During matching, the cosine similarity between the feature vector and the center of each category is calculated, and the highest result is selected; similarity less than 0.6 is judged as "neutral". The recommendation accuracy rate reaches 91.2% (recommendation accuracy = Test set: 2000 comic scene samples; matching criteria: dual constraints of scene label + sentiment label; statistical results: 1824 correct → 91.2%. For scenes not covered by the knowledge base, a diffusion model was used for real-time generation, and the spectral similarity of the generated sound effects matched the target scene by more than 87%.

[0087] Quality Optimization and Output Module This module constructs a three-tiered quality control system: Level 1 control uses objective indicators (including speech intelligibility PESQ ≥3.8, emotion matching ≥85%, and background noise ≤-60 dB); Level 2 control employs a pre-trained audio quality evaluation model (based on a CNN-LSTM architecture, with an evaluation accuracy of 92.5%); Level 3 control supports a manual review interface, allowing for the labeling and feedback of automatically generated audio, forming a closed-loop optimization. The output unit supports multi-format export (MP3, WAV, AAC) and provides an audio synchronization information file (JSON format) containing the timestamp mapping relationship between speech segments and comic panel layouts (synchronization accuracy ≤50ms), meeting the precise alignment requirements of post-production.

[0088] System Implementation Process Test conditions: Hardware configuration GPU: NVIDIA A100 (40GB) × 2 CPU: Intel Xeon Platinum 8358 Memory: 256GB Storage: 4TB NVMe SSD Software environment System: Ubuntu 20.04 Framework: PyTorch 2.0 + HuggingFace Transformers Image processing: OpenCV 4.6 Audio processing: Librosa 0.10 Batch processing capability Single-second processing: 3 frames 200-page comic: 45 minutes Overall latency: ≤6.5 seconds / storyboard The system processing flow consists of five key steps: Step 1, multimodal data acquisition and preprocessing (approximately 1.2 seconds), completing image denoising, text detection, and audio standardization; Step 2, cross-modal feature extraction (approximately 1.5 seconds), processing image, text, and reference audio features in parallel; Step 3, feature fusion and semantic mapping (approximately 0.8 seconds), generating structured audio generation instructions; Step 4, speech and sound effect synthesis (approximately 2.3 seconds), dynamically allocating TTS and sound effect generation resources according to scene requirements; Step 5, quality optimization and output (approximately 0.7 seconds), ensuring output quality through a three-level quality control system. The total processing time is controlled within 6.5 seconds, supporting batch processing capabilities of 3 scenes per second.

[0089]

Complete Data Stream Example

[0090] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for generating audio for audio comics based on multimodal data.

[0091] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0092] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for intelligent audio processing of audio comics based on multimodal AI, characterized in that, Includes the following steps: S1, acquire the comic panel image to be processed and the corresponding text dialogue data, perform noise reduction and enhancement processing on the comic panel image to obtain a preprocessed image, and identify the text region in the preprocessed image to extract visual position information. S2, respectively encode the preprocessed image and the text dialogue data to obtain image feature vector and text feature vector, use the visual position information to perform cross-modal fusion of the text feature vector and the image feature vector, and combine the reading time sequence of the comic panel to generate a three-dimensional emotional feature vector containing emotional intensity, spatial position and semantic priority. S3. Based on the three-dimensional emotion feature vector, determine the speech synthesis parameters and scene sound effect matching strategy, use the pre-trained speech generation model in combination with the speech synthesis parameters to generate character speech data, and retrieve scene sound effect data from the sound effect library or synthesize it through the generation model according to the scene sound effect matching strategy. S4, perform quality assessment and synchronization calibration on the character voice data and the scene sound effect data, and splice the calibrated audio data according to the reading sequence of the comic panel to generate a target audio file synchronized with the comic panel screen.

2. The method as described in claim 1, characterized in that, S1 includes: S11, a bilateral filtering algorithm is used to perform pixel-level denoising on the comic panel image to eliminate high-frequency noise interference, and the multi-scale decomposition technique based on Retinex theory is used to perform contrast enhancement processing on the denoised image to obtain an enhanced image with clear details. S12, the text detection network combining CTPN and Mask R-CNN models is used to scan and locate the dialog box region in the enhanced image, extract visual position information containing the normalized coordinates of the center point of the text box, and convert the visual position information into an embedding vector for subsequent feature fusion.

3. The method as described in claim 2, characterized in that, S12 includes: S121, perform text line detection on the enhanced image using the CTPN model to generate a candidate text region sequence, and input the candidate text region sequence into the Mask R-CNN model for instance segmentation to accurately define the dialog box boundary; S122, calculate the normalized coordinates of the center point of the character's facial region based on the dialog box boundary, and map the normalized coordinates into a two-dimensional spatial position vector, which is then output to the cross-modal fusion stage as a core component of the visual position information.

4. The method as described in claim 1, characterized in that, S2 includes: S21, feature extraction is performed on the preprocessed image based on the improved CLIP model. The stylized elements unique to comics are captured through the newly added halftone texture feature extraction branch, generating an image feature vector containing global semantics and local texture. S22, The text dialogue data is encoded using the BERT-Base model that incorporates visual position encoding, and the visual position information is converted into an embedding vector and superimposed on the text semantic features to generate a text feature vector; S23, construct a multi-head attention mechanism, calculate the association weight between the text feature vector as the query vector and the image feature vector as the key vector and value vector, and generate multi-head attention output features; S24, the multi-head attention output features are input into the gated recurrent unit (GRU), and temporal encoding is performed according to the reading order of the comic panel. The hidden state containing temporal context information is output, and the hidden state is concatenated with the multi-head attention output features and then passed through a linear projection layer to generate a cross-modal feature vector. S25, Based on the cross-modal feature vector, construct a three-dimensional emotion feature vector: V_emotion=[EI, SP_x, SP_y, Priority], Among them, the emotional intensity EI is obtained by predicting the amplitude of facial muscle movements and the dot density through CNN regression, the spatial positions SP_x and SP_y are derived from the visual position information, and the semantic priority is obtained by normalizing the emotional saliency calculated by the attention mechanism.

5. The method as described in claim 4, characterized in that, S23 includes: S231, the text feature vector is multiplied by the query weight matrix of the i-th attention head to obtain the query vector, and the image feature vector is multiplied by the key weight matrix and value weight matrix of the i-th attention head to obtain the key vector and value vector respectively. S232, calculate the output of the i-th attention head according to the single-head attention weight formula, wherein the single-head attention weight formula is: Where, d k It is a single-head dimension; S233, the parallel computation process is executed to learn local facial expressions, global scene, text tone and image focus features respectively, the outputs of all attention heads are concatenated and linearly projected through the output weight matrix to obtain the multi-head attention output features.

6. The method as described in claim 4, characterized in that, S24 includes: S241, the attention features of the scene at time t and the hidden state of the previous time step are input into the update gate and the reset gate, respectively. The update gate control signal is calculated according to the update gate formula, and the reset gate control signal is calculated according to the reset gate formula. The update gate formula is as follows: ； The reset gate formula is: ； S242, the hidden state of the previous time step is weighted and filtered using the reset gating signal, and candidate hidden states are calculated by combining the attention features of the current time step: ； S243, based on the updated gating signal, linear interpolation is performed to fuse the hidden state of the previous time step with the candidate hidden state, and the temporal output hidden state at time t is generated according to the output hidden state formula to represent the dynamic narrative logic between comic panel scenes. The output hidden state formula is: 。 7. The method as described in claim 1, characterized in that, S3 includes: S31, analyze the emotional intensity EI and semantic priority in the three-dimensional emotional feature vector, dynamically adjust the speech rate fluctuation range and pitch change amplitude in the speech synthesis parameters, and set the spatial reverberation parameters that are dynamically adjusted based on the spatial location and the early reflection coefficient. S32, adopting a GAN-enhanced end-to-end TTS architecture, injects the speech synthesis parameters into the style embedding layer of the decoder, extracts the speaker style features of the reference audio through the StyleGAN2 network and generates style vectors to control timbre and prosody, and generates the intermediate Mel spectrum; S33, the intermediate Mel spectrum is input to the MelGAN vocoder to perform waveform synthesis, and the output is character voice data with specific emotional expression and spatial positioning effect; S34, the Mask R-CNN segmentation network is used to identify the visual semantic parameters in the preprocessed image to determine the scene classification label, and a three-dimensional emotion space model including valence, arousal and dominance is constructed based on the three-dimensional emotion feature vector. S35, calculate the cosine similarity between the feature vector formed by the scene classification label and the three-dimensional emotional space model and the center vector of each category in the sound effect knowledge base. When the highest similarity is greater than or equal to the set threshold, retrieve the scene sound effect data with the highest matching degree from the sound effect sample knowledge base. S36. When the highest similarity is less than the set threshold and is determined to be a neutral or non-covered scene in the knowledge base, the Diffusion Model is adopted with the visual semantic parameters and the three-dimensional emotion space model as conditional inputs.

8. The method as described in claim 1, characterized in that, S4 includes: S41, perform first-level objective index detection to verify whether speech clarity PESQ is greater than or equal to the clarity threshold, emotion matching degree is greater than or equal to the matching threshold, and background noise is less than or equal to the noise threshold, and perform second-level automated quality scoring through a pre-trained audio quality assessment model based on CNN-LSTM architecture; S42, Generate a JSON-formatted synchronization information file containing the mapping relationship between voice segments and comic panel timestamps based on the reading sequence of the comic panel, and use a dynamic time warping algorithm to align the character voice data and the scene sound effect data at the millisecond level; S43 seamlessly splices the aligned audio data according to the scene sequence and exports it as the target audio file.

9. An audio intelligent processing system for audio comics based on multimodal AI, characterized in that, Includes the following steps: The image and text preprocessing module is used to acquire the comic panel image to be processed and the corresponding text dialogue data, perform noise reduction and enhancement processing on the comic panel image to obtain a preprocessed image, and identify the text region in the preprocessed image to extract visual position information. The emotional feature fusion module is used to encode the preprocessed image and the text dialogue data to obtain image feature vectors and text feature vectors respectively, use the visual position information to perform cross-modal fusion of the text feature vectors and the image feature vectors, and combine the reading time sequence of the comic panel to generate a three-dimensional emotional feature vector containing emotional intensity, spatial position and semantic priority. The speech and sound effect generation module is used to determine the speech synthesis parameters and scene sound effect matching strategy based on the three-dimensional emotion feature vector, generate character speech data by combining the pre-trained speech generation model with the speech synthesis parameters, and retrieve scene sound effect data from the sound effect library or synthesize it through the generation model according to the scene sound effect matching strategy. The audio timing synthesis module is used to perform quality assessment and synchronization calibration on the character voice data and the scene sound effect data, and to splice the calibrated audio data according to the reading sequence of the comic panel to generate a target audio file synchronized with the comic panel screen.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as claimed in any one of claims 1-8.

Images