A method and system for new media multi-modal content production based on artificial intelligence

Abstract

The application discloses a new media multi-modal content production method and system based on artificial intelligence, and relates to the technical field of new media content generation, and comprises the following steps: S001, a nanosecond time reference is established, a time sequence texture band in a continuous time dimension is constructed by collecting voice beat signals, image frame edge signals and action key point signals, and is used for subsequent time alignment reference of multi-modal signals. Through the construction of the nanosecond time reference and the delay channel graph, the application realizes accurate alignment and drift tracking of the multi-modal signals, improves the time sequence stability and fault tolerance, ensures the rhythm consistency and expression coherence of the voice, image and action in the generation process, and enhances the perception coordination and semantic accuracy of the content.

Description

Technical Field

[0001] This invention relates to the field of new media content generation technology, specifically to a method and system for producing new media multimodal content based on artificial intelligence. Background Technology

[0002] Artificial intelligence-based multimodal content production for new media refers to the process of using artificial intelligence algorithms to comprehensively analyze and intelligently generate various information modalities (such as text, images, audio, video, and motion data) to achieve automated creation and optimization of new media content. This method utilizes multimodal data fusion technology to extract features and semantically align data from different sensory sources. It leverages deep learning models to understand the semantic relationships between different modalities and automatically generates integrated media works with consistent style and logical coherence based on target content requirements. Information processing plays a central role in this process. Through data acquisition, signal preprocessing, noise filtering, feature extraction, semantic parsing, and multi-layer coding, raw multimodal data is transformed into structured information that can be understood and generated by the model, enabling the system to achieve high-precision information flow and semantic mapping between different modalities. For example, the system can automatically generate accompanying images, videos, or audio descriptions based on an input text description, achieving cross-modal creation from text to visual and auditory elements. This technology breaks through the limitations of traditional single-modal content production, significantly improving content production efficiency and intelligence, and has broad application value in fields such as new media marketing, short video creation, digital advertising design, and virtual human content generation.

[0003] The existing technology has the following shortcomings: In existing technologies, AI-based new media multimodal content production systems typically rely on the synchronous acquisition and fusion of audio, image, and motion streams to achieve temporal consistency in content generation. However, when sampling delays, cache refresh differences, or network clock offsets exist during the real-time inflow of multi-source data, false time alignment signals can easily form within the model, resulting in pseudo-synchronous resonance. This phenomenon causes the model to mistakenly believe that the multimodal data has achieved complete synchronization during timing judgment, triggering incorrect fusion operations and leading to a delay illusion in the generated footage during dynamic playback. This illusion manifests as a visually coordinated appearance of lip movements, speech rhythm, and motion beats, but in reality, there are microsecond to millisecond-level offsets. As the content length increases, these offset errors are dynamically amplified, causing distorted visual rhythm and semantic expression, severely impacting the stability and credibility of the multimodal generation results.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for producing new media multimodal content based on artificial intelligence, so as to solve the problems in the background art mentioned above. To achieve the above objectives, the present invention provides the following technical solution: a method for producing new media multimodal content based on artificial intelligence, comprising the following steps:

[0006] S001, establish a nanosecond-level time reference by collecting speech beat signals, image frame edge signals and action key point signals to construct a temporal texture band in the continuous time dimension for subsequent multimodal signal time alignment reference; S002, based on the temporal distribution characteristics in the temporal texture band, constructs the energy progression sequence of speech signals, image signals and action signals, transforms the arrival order of multimodal signals into energy ladders, and identifies peak segments with temporal misalignment between signals to generate pseudo-synchronous peaks; S003, embeds reverse time markers at the leading and trailing edges of each pseudo-synchronization peak segment to record the time drift direction and amplitude between signals, forming a set of reverse anchor points with time traceability capability; S004, based on the reverse anchor point set, rearranges the time stream, and generates a delay channel diagram containing buffers and detour areas by adjusting the trigger order and beat interval, giving the time stream multi-path redundancy capability; S005, based on the structural relationship of the delay channel diagram, sequentially activates the micro-periodic misalignment choreography engine, shadow frame insertion mechanism, phase conjugate diversion structure, voice breathing rhythm grid structure and motion hysteresis valve chain structure, so that the time series between voice signal, image signal and motion signal maintain a synchronous and stable state.

[0007] Preferably, step S001 includes:

[0008] A unified time reference in nanoseconds is constructed using a high-frequency sampling clock, and the voice input device, image acquisition device and motion capture device are configured to synchronize clocks. A unified time trigger signal is used to ensure that the sampling start points of each device are strictly consistent in time.

[0009] During the data acquisition process, the speech signal is sampled based on the beat of sound pressure fluctuation, the image signal is sampled based on the frame edge change, and the motion signal is sampled based on the extreme value change point of joint position. Each sampling point is assigned a unique timestamp and assigned to a unified time coordinate axis.

[0010] Based on whether there are changes in speech rhythm, image frame edge, or key points of action at each time point, event markers are established and classified, and changes in signal response intensity and occurrence order are recorded to form a sequence of perceptual events.

[0011] A complete temporal texture band is constructed based on the perceived event sequence. The high-frequency response area is extracted and summarized to form a temporal structure with temporal readability and continuous triggering pattern, which is used to guide the time alignment of subsequent multimodal signals.

[0012] Preferably, during the establishment of event markers, a dynamic weight adjustment strategy is introduced for each type of signal. Based on the interleaving frequency of multimodal signals at the same time point, the expression intensity of the signal at the interleaving moment is enhanced, thereby improving the accuracy of the perception event sequence in depicting the time triggering relationship.

[0013] Preferably, step S002 includes:

[0014] On the established temporal texture band, the frequency of occurrence, amplitude of change and continuous trigger span of speech signal, image signal and action signal in each time interval are extracted segment by segment to generate time feature points with three-dimensional energy response values.

[0015] Based on the arrangement of time feature points on the time axis, the energy change trends of each signal are sampled and superimposed to form a continuous energy ladder consisting of multiple energy increase segments and energy decay segments, thus constructing an energy progression sequence under a unified time reference.

[0016] The peak time positions of speech signals, image signals and action signals in the energy progression sequence are compared, the peak time difference and energy amplitude difference are calibrated, the trigger misaligned segments are identified, and a peak index table containing time offset parameters is generated.

[0017] Based on the peak index table, extract the time-off peak groups that appear periodically or quasi-periodically, match the peak pairs to generate pseudo-synchronous peak columns with time positioning and energy characteristics, and establish a one-to-one correspondence with the time-series texture band.

[0018] Preferably, when constructing an energy progression sequence under a unified time reference, a smoothing process is used to transition and adjust the energy abrupt change regions between adjacent energy segments, so that the energy changes of speech signals, image signals, and motion signals on the time axis form a continuous and stable trapezoidal distribution.

[0019] Preferably, step S003 includes:

[0020] Based on the energy progression sequence, the leading edge and trailing edge boundaries of pseudo-synchronous peak segments are determined one by one. By analyzing the changes in the energy curves of speech signals, image signals, and action signals, the time points at which the signals begin to deviate and realign are extracted as segment boundaries.

[0021] Time markers with direction information and time interval parameters are embedded at each leading and trailing edge boundary. The drift direction and time offset are recorded by comparing the peak signal energy with the standard reference position.

[0022] Each time marker is bound to its corresponding timeline position to build a set of anchor points with time as the main index, and a chain connection is established according to the order of peak segments to form a complete time traceability structure.

[0023] By jointly mapping the anchor point set with the energy progression sequence and pseudo-synchronous peaks, a multi-layered data structure containing drift direction, energy trend, and temporal position is established to support subsequent time re-arrangement and synchronization control.

[0024] Preferably, step S004 includes:

[0025] Based on the set of reverse anchor points, the offset trend and distribution density of the signal on the time axis are extracted, and a set of time segments containing time boundaries and directional features is constructed as the basic framework for rearrangement.

[0026] Based on the set of time segments, the triggering order and beat interval of the signals in each segment are adjusted to maintain the continuity and consistency of the rhythmic relationship between the signals and to construct a rhythmically coordinated local time flow structure.

[0027] The locally reconstructed paragraphs are globally spliced ​​together, and buffer and detour areas are inserted between the paragraphs to construct a delay path graph with delay absorption and path substitution capabilities.

[0028] Based on the structural relationship of the delay channel diagram, the path number is bound to the signal triggering command to establish a dynamic channel mapping, which enables the time flow to have multi-path redundancy and rhythm stability.

[0029] Preferably, the buffer area is used to absorb the time error caused by the triggering of the beat adjustment. The time length is dynamically set according to the drift difference between adjacent segments. The detour area is used to build a delay branch for the signal. After the adjustment is completed through an independent channel outside the main path, it returns to the main channel to achieve stable rhythm output of multimodal signals under complex time offset conditions.

[0030] Preferably, step S005 includes:

[0031] Based on the signal path segments in the delay channel diagram, the micro-periodic misalignment choreography engine is activated to correct and adjust the micro-time rhythm of the voice signal, image signal, and motion signal.

[0032] To address rhythmic gaps and overlapping areas in image signals during frame rate changes, a shadow frame interpolation mechanism is activated to perform image transition compensation and frame content coordination.

[0033] To address the phase shift of the speech signal in the delay path, a phase conjugate diversion structure is activated to adjust the start time of the speech waveform to achieve beat alignment.

[0034] By using a voice breathing rhythm grid structure and an action hysteresis valve chain structure, the rhythm synchronization and delayed release of action signals are achieved, so that each signal maintains a stable output rhythm relationship.

[0035] A system for producing new media multimodal content based on artificial intelligence includes a timing reference module, a misalignment detection module, a drift marking module, a channel arrangement module, and a synchronization control module. The timing reference module establishes a nanosecond-level timing reference by acquiring speech beat signals, image frame edge signals, and action key point signals to construct a temporal texture band in a continuous time dimension, which is used as a time alignment reference for subsequent multimodal signals. The misalignment detection module constructs an energy progression sequence of speech signals, image signals, and action signals based on the temporal distribution characteristics in the temporal texture band. It converts the arrival order of multimodal signals into energy ladders and identifies peak segments with temporal misalignment between the signals, generating pseudo-synchronous peak sequences. The drift marker module embeds reverse time markers at the leading and trailing edges of each pseudo-synchronization peak segment to record the direction and amplitude of time drift between signals, forming a set of reverse anchor points with time traceability capability. The channel orchestration module, based on the reverse anchor point set, re-orchestrates the time stream and generates a delay channel diagram containing buffers and detour areas by adjusting the trigger order and beat interval, giving the time stream multi-path redundancy capability. The synchronization control module, based on the structural relationship of the delay channel diagram, sequentially activates the micro-periodic misalignment choreography engine, shadow frame insertion mechanism, phase conjugate diversion structure, voice breathing rhythm grid structure, and motion hysteresis valve chain structure, so that the time series of voice signal, image signal, and motion signal maintain a synchronous and stable state.

[0036] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0037] This invention establishes a nanosecond-level time reference to ensure consistent acquisition of multi-source data at the microscopic time scale. By constructing temporal texture bands and energy progression sequences, it accurately identifies the temporal offset regions of multimodal signals, extracts pseudo-synchronization peak segments, and then embeds directional reverse time markers at key time points to form a traceable time anchor structure, achieving comprehensive perception and quantitative expression of signal drift paths. Through the rearrangement of the time flow, a delay channel map with buffering and detour capabilities is constructed, enabling multimodal signals to dynamically adapt to different paths, significantly enhancing the system's fault tolerance to complex temporal fluctuations. Finally, combined with a refined beat control mechanism and a multi-layered synchronization structure, it ensures that speech, image, and motion signals maintain a highly consistent output rhythm and coordinated expression throughout the long-term content generation process, significantly improving the stability and credibility of multimodal generated content in terms of perceptual coherence, semantic accuracy, and user experience. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0039] Figure 1 This is a flowchart of the method of the present invention.

[0040] Figure 2 This is a schematic diagram of the modules of the present invention. Detailed Implementation

[0041] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0042] This invention provides, for example Figure 1 The method for producing new media multimodal content based on artificial intelligence, as shown, includes the following steps:

[0043] S001, establish a nanosecond-level time reference by collecting speech beat signals, image frame edge signals and action key point signals to construct a temporal texture band in the continuous time dimension for subsequent multimodal signal time alignment reference; To address the issue of misjudgment during fusion caused by slight temporal shifts in multi-source signals, a high-precision time reference needs to be established. This is achieved by constructing a continuous and traceable temporal texture structure, which serves as the foundation for subsequent multimodal signal alignment and collaborative generation. The specific steps are as follows: A nanosecond-level time reference is constructed using a high-frequency sampling clock, synchronizing the voice input device, image acquisition device, and motion capture device used for data acquisition. By configuring a unified time trigger signal, each device receives a precise and consistent start pulse at the initial moment of data acquisition, ensuring that the sampling start points of all sensing devices are strictly consistent in time. Subsequently, during the sampling process, each data stream is recorded at a set fixed time interval. The sampling of voice signals is segmented based on the beat of sound pressure fluctuations, the image signal is segmented based on the rising and falling edges of the frame, and the motion signal is segmented based on the extreme value changes of joint positions. Each sampling point is assigned a unique timestamp and uniformly categorized into a time coordinate axis with nanoseconds as the smallest unit. In this way, the parallel recording of the three types of signals in the same time dimension is achieved, providing a highly granular basic coordinate reference for subsequent synchronization processing.

[0044] Based on timestamps, the speech beat signal, image frame edge signal, and motion key point signal are structurally integrated, and the timestamp information of the three types of signals is fused and encoded along a nanosecond-level time axis. Specifically, each nanosecond time unit is scanned, and if any one or more of the following occur at that time point: speech beat change, image frame edge change, or motion key point change, an event marker is established and categorized at that time point. This categorization marker uses time as the core dimension, recording the response intensity changes and occurrence order of each type of perceived signal, forming a perceptual event sequence arranged in chronological order. To avoid low-intensity signals being marginalized in high-frequency sampling, a dynamic weight adjustment strategy is introduced for each type of signal to strengthen its expression weight at the moment of multimodal interleaving. After this processing, the signal composition at each time point has a clear event label, which can clearly describe the triggering situation of different signals at that time node and their relative temporal relationship.

[0045] Based on event-marked sequences, a complete temporal texture structure is constructed across the entire timeline. This structure uses nanosecond time intervals as the horizontal axis and the triggering states of multimodal signals as the vertical axis, visually arranging the signal composition at each time point as texture strips. During construction, local scanning is first performed in time intervals, defining areas with frequent continuous signal occurrences as high-density bands and areas with scattered and discontinuous signals as low-density bands. The timeline is then segmented based on these density variations, extracting the main segments representing strong signal responses. Within each main segment, the signal trajectory is further smoothed, ensuring that the highs and lows of speech beats, edge changes in image frames, and sudden speed changes at key action points all form continuous trend curves. These trend curves are superimposed on a unified timeline through logical mapping relationships, forming a hierarchical and clearly structured temporal texture strip. This not only possesses temporal readability but also demonstrates the continuity of triggering patterns, providing a unified dynamic benchmark for subsequent multimodal data fusion.

[0046] The constructed temporal texture band serves as a time alignment reference for multimodal signals, guiding downstream signal fusion and sequence rearrangement processes. In subsequent processing, each newly added sensing signal, before entering the fusion stage, must be compared and matched with the main segment time in the temporal texture band to determine if it falls within a strong response range, and its time position is corrected accordingly. If a signal falls into a high-density area of ​​the texture band, it indicates high multimodal commonality at the current time point and can directly participate in the alignment operation; if it is in a low-density area, fine-tuning is required based on the temporal extension trend of adjacent main segments to ensure a reasonable trigger offset between it and the main segments in time. Through this matching and correction mechanism, the temporal distribution of all sensing signals achieves high consistency under the constraints of the texture band, providing a highly reliable and high-resolution temporal foundation for subsequent energy sorting, signal alignment, and co-generation processes. This fundamentally avoids alignment misjudgments and fusion distortion caused by inconsistent sampling starting points, time drift, or trigger jitter.

[0047] S002, based on the temporal distribution characteristics in the temporal texture band, constructs the energy progression sequence of speech signals, image signals and action signals, transforms the arrival order of multimodal signals into energy ladders, and identifies peak segments with temporal misalignment between signals to generate pseudo-synchronous peaks; After establishing the temporal texture band, to further reveal the true correlation between multimodal signals in the temporal dimension, it is necessary to construct an energy progression sequence of speech signals, image signals, and action signals based on the temporal distribution characteristics in the temporal texture band. This converts the trigger intensity of each modal signal at different time points into a measurable energy ladder, thereby identifying temporal misalignment regions and generating pseudo-synchronization peaks, laying the foundation for subsequent drift localization and temporal rearrangement. The specific steps are as follows: On the established temporal texture strip, the temporal distribution of speech, image, and motion signals is analyzed segment by segment. By traversing the nanosecond-level time scale of the texture strip, the frequency of occurrence, amplitude of variation, and duration of consecutive triggering of the three types of signals within each time interval are extracted. At the speech signal level, the periodic changes in sound pressure intensity are analyzed, distinguishing and recording the energy rise and fall segments of two adjacent beats. At the image signal level, the continuity of frame edge changes and the strength of edge contrast are monitored to determine the energy concentration areas of motion in the image. At the motion signal level, the velocity change amplitude of key points in motion is calculated in spatial coordinates to identify energy surges during body posture transitions or rapid joint movements. Subsequently, the intensity characteristics of these three types of signals within the same time interval are uniformly mapped onto the time axis of the temporal texture strip to form a set of composite temporal feature points containing three-dimensional energy parameters. Each feature point simultaneously possesses energy response values ​​for speech, image, and motion, providing a foundation for constructing multimodal energy progression relationships.

[0048] After obtaining the temporal feature points, their energy change trends on the time axis are arranged vertically to form a continuous energy gradient structure. In this process, using the temporal sequence of the texture band as the main thread, the energy values ​​of each feature point are continuously sampled and superimposed, so that temporally adjacent signal intensities form an increasing or decreasing energy sequence. When the energy of the speech signal beat increases simultaneously with changes in the image frame edge and an increase in the acceleration of the action keypoint, this time segment is defined as the energy progression region; if any of the three types of signals experiences delayed triggering or energy decrease, it is marked as the energy decay region. In this way, multiple continuous energy gradients can be formed on the time axis, each representing a coordinated change cycle of a multimodal signal. To maintain the continuity between energy gradients, the transition regions between adjacent energy segments are smoothed, extending the signal energy abrupt changes into a stable trapezoidal distribution, thus forming an energy progression sequence of speech signals, image signals, and action signals under a unified time reference.

[0049] After the energy progression sequence is constructed, the identification and calibration of time-displacement regions begin. By comparing the triggering order of different signal types in the energy progression sequence, the consistency of the energy peak positions of the speech signal, image signal, and action signal within the same time period is detected. When the energy peak of the speech signal appears before the energy peak of the image frame, while the energy peak of the action signal is later than that of the image signal, it indicates a signal arrival delay within that time period. To further determine the nature of the delay, a differential time scan is performed on the energy curve of this region to calibrate the specific time difference between the occurrence of the maximum and minimum values ​​of energy change, and the corresponding energy offset is recorded. When the time difference between the energy peaks of the three types of signals exceeds a set threshold, the region is defined as a time-displacement segment. In each time-displacement segment, the ratio of the energy amplitude and the ratio of the time difference between the speech signal, image signal, and action signal are recorded, thereby generating a peak index table containing specific time offset characteristics, providing data for pseudo-synchronization identification.

[0050] Based on the calibrated time-displacement segments, all pseudo-synchronization peak groups that appear periodically or quasi-periodically on the time axis are extracted to form a complete pseudo-synchronization peak sequence. In this process, using the energy progression sequence as a reference frame, peak pairs with similar energy distribution characteristics but opposite time offset directions are paired to confirm their relative delay relationship. When a pair of peaks repeats in multiple time periods, and its energy peak difference and time offset difference remain stable, it is identified as a fixed peak group in the pseudo-synchronization peak sequence. In the generated pseudo-synchronization peak sequence, each peak group represents a temporal erroneous resonance phenomenon of a multimodal signal. To facilitate subsequent reverse identification and time correction, the pseudo-synchronization peak sequence is indexed according to time order and corresponds one-to-one with the original time-series texture band, so that each pseudo-synchronization peak has a unique time positioning identifier and energy calibration parameters. Through the above process, a complete process was established, from time distribution characteristics to energy mapping, from energy progression to misalignment detection, and from misalignment identification to pseudo-synchronous peak generation. This enabled the precise revelation of the time offset of multimodal signals, providing a high-precision basis for the next stage of time drift localization and inverse marker implantation.

[0051] S003, embeds reverse time markers at the leading and trailing edges of each pseudo-synchronization peak segment to record the time drift direction and amplitude between signals, forming a set of reverse anchor points with time traceability capability; After generating pseudo-synchronization peaks based on the energy progression sequence, in order to further identify and trace the specific time drift path of the multimodal signal, it is necessary to perform boundary calibration on each pseudo-synchronization peak segment. By embedding directional time markers, a set of reverse anchor points that can be used for subsequent rearrangement and error correction is constructed, thereby achieving accurate location and tracking of the misalignment root cause. The specific steps are as follows: For the established pseudo-synchronization peak sequence, the leading and trailing edge trends of each peak segment in the time dimension are analyzed one by one to clarify the starting and ending points of signal misalignment. In this process, based on the energy progression sequence constructed in the previous stage, the energy curves of the speech, image, and motion signals corresponding to each peak segment are finely decomposed. The time points when various signals begin to deviate from the coherent trend are extracted as the segment leading edge, and the time points when various signals re-align are extracted as the segment trailing edge. These two time points represent the critical points where signal drift begins and ends, respectively, and are key evidence for identifying the time error fluctuation cycle. During the extraction process, to ensure the accuracy of the time boundaries, a sliding time window is used to compare the energy response times between each signal point by point, filtering out short-term fluctuations and interference peaks, retaining boundary data with clear continuous trends, and finally determining the start and end time range of each pseudo-synchronization segment.

[0052] At the leading and trailing edges of each segment, directional time stamps are embedded to record the direction and magnitude of signal shift. At each boundary time point, the speech, image, and motion signals are sampled separately, and the relative position change of their current energy peaks is calculated and compared with the expected peak position under standard coordinated conditions. Based on this comparison, the direction of drift is determined; for example, if the speech signal triggers earlier while the image signal lags, the drift direction can be determined to be shifting from speech-dominated to image-dominated. The magnitude of the drift is then recorded based on the actual time interval between peak positions, precisely measured in nanoseconds. This directional information and time interval data are encapsulated into directional time stamps, embedded at the leading and trailing edges of segments. This ensures that the boundaries of each pseudo-synchronous segment not only have time positioning capabilities but also indicate the drift trend, providing directional signals for subsequent reverse correction.

[0053] Embedded time markers are indexed and bound to their corresponding timeline positions to construct a set of anchor points with time-tracking capabilities. A sequential arrangement of time anchor points is established across the entire temporal texture band, using time as the primary index. Each anchor point consists of four items: segment identifier, signal type, drift direction, and time offset value, linked in a chain according to the order of peak segment appearance. To enhance the coherence and resolvability of the anchor point set in subsequent processing, the drift direction change trends between adjacent anchor points are categorized. Anchor points with the same direction and similar amplitude are grouped into a single time drift segment, and an order relationship is established within each segment. Through this structure, the offset behavior of each type of signal in multiple pseudo-synchronization regions forms a coherent historical trajectory, constructing a traceable, measurable, and controllable network of time migration paths. This allows the system to trace back to any drift starting point and analyze the source and propagation process of signal misalignment.

[0054] By jointly mapping the set of reverse anchor points with the energy progression sequence and pseudo-synchronous peaks, a time behavior model with a logical closed-loop relationship is formed, laying a multi-dimensional data foundation for subsequent time stream rearrangement. In this joint mapping structure, the pseudo-synchronous peaks serve as the central node, with each peak segment acting as the core carrying area of ​​the anchor point set. Simultaneously, the energy ladder constructed in the previous step provides background support, marking the energy trend type (rising segment, plateau segment, falling segment) of the anchor points. Combined with anchor point direction data, the offset path trend of each signal is recorded. This multi-layered mapping structure not only gives reverse anchor points a logical affiliation but also incorporates the contextual energy environment of the anchor points into the overall consideration, improving the adaptation accuracy and correction specificity of drift data. Based on this structure, in subsequent time adjustment and channel construction, the adjustment of each signal position will have contextual constraints, historical references, and logical continuity, enabling the drift problem not only to be located and tracked but also to maintain the stability and self-consistency of the correction path during dynamic operation. Through this series of embedding, building, and mapping processes, a set of reverse anchor points with directionality, precision, and traceability is finally constructed, providing a solid timing foundation for subsequent time orchestration and synchronization adjustments.

[0055] S004, based on the reverse anchor point set, rearranges the time stream, and generates a delay channel diagram containing buffers and detour areas by adjusting the trigger order and beat interval, giving the time stream multi-path redundancy capability; After constructing a set of reverse anchor points with time-tracking capabilities, to address the timing imbalance caused by the time-dimensional shift of multimodal signals, it is necessary to rearrange the original time stream using the anchor point set as a reference. This involves constructing delay structures and establishing channel paths with buffering and detour capabilities, thereby giving the time stream multi-path redundancy and improving its adaptability to complex time drift phenomena. The specific steps are as follows: Based on the formed set of reverse anchor points, the offset trend and anchor point distribution density of each signal type on the time axis are extracted to construct a preliminary time-series partitioning structure. This partitioning structure uses the time axis as a continuous reference, dividing it into multiple segments, each containing a set of continuously distributed reverse anchor points. During the partitioning process, areas with dense anchor points are marked as offset-prone segments, indicating a trend of concentrated signal misalignment that requires focused arrangement in subsequent processing. Simultaneously, directional markers are added to each segment, and based on the time drift direction information in the anchor points, it is determined whether the main signal offset trend in the current segment is advanced, lagging, or cross-changing. Through this method, a set of time-series segments with time boundaries, anchor point attributes, and directional characteristics is obtained, serving as the basic framework for re-arrangement.

[0056] Based on a set of time sequence segments, the signal triggering order and beat intervals within each segment are reconstructed and adjusted to form a rhythmically controllable multimodal time flow layout. Specifically, for signal sequences exhibiting time drift within the same segment, their triggering start time within that segment is adjusted according to the drift direction and offset magnitude marked by anchor points, resulting in a fine-grained offset relative to their original position on the time axis. This offset must meet the following conditions: first, maintaining the continuity of the original relative rhythmic relationship between signals to prevent new misalignments after adjustment; second, for signals with large drift magnitudes, a gradual approach is used to adjust their triggering time to avoid abrupt changes caused by a large one-time displacement. In addition to adjusting the start time, the beat intervals between adjacent triggering events also need to be fine-tuned to maintain logical smoothness and perceptual consistency under the new time sequence. Through this method, the signal flow in each segment is reordered and intervals are configured, constructing a locally optimal triggering rhythm structure and laying a stable foundation for the overall time flow reorganization.

[0057] After completing the local time reconstruction, all segments are globally stitched together, embedding structural regions with buffering and bypass capabilities to construct the overall delay channel diagram. Buffer regions are micro-time bands inserted between two segments to absorb residual time errors caused by trigger rhythm adjustments. The duration of these regions is dynamically set based on the drift differences between adjacent segments, varying in length to achieve seamless transitions between different segments. Detour regions are alternative path regions constructed when large-scale time drift in the original time stream cannot be corrected through single-point adjustments. Within detour regions, delay branches are constructed for specific signal types, allowing them to complete independent adjustments outside the main time path before returning to the main channel, thus avoiding the disruptive impact of concentrated drift on the overall rhythm. The entire delay channel diagram, through the combination of the main path, buffer regions, and detour branches, forms a multi-channel parallel, multi-rhythm fusion time stream distribution structure, providing a flexible and stable time execution channel for multimodal data.

[0058] By associating the time structure parameters of the delay channel graph with the signal control strategy, a dynamically callable channel mapping relationship is established to support the subsequent time stream operation. In this stage, based on the path planning in the delay channel graph, the channel path to be used by each type of signal in each time period is numbered, and this number is bound to the signal's trigger command, enabling the signal to automatically match different delay paths at different time periods. During signal execution, based on the mapping state between the current time and the channel graph, it is determined whether the signal should enter the main channel, the buffer, or detour through a branch, and its specific execution timing is adjusted accordingly. In this way, the entire time stream is no longer a linear, single-path execution, but a composite structure with multi-path redundancy, adaptive adjustment, and rhythm stability characteristics under the drive of reverse anchor points. This provides high flexibility and dynamic adaptability for subsequent synchronization control and error correction, ensuring that various signals maintain a consistent output state even under complex time drift backgrounds during multimodal content production.

[0059] S005, based on the structural relationship of the delay channel diagram, sequentially activates the micro-periodic misalignment choreography engine, shadow frame insertion mechanism, phase conjugate diversion structure, voice breathing rhythm grid structure and motion hysteresis valve chain structure to keep the time series of voice signal, image signal and motion signal in a synchronous and stable state. After establishing the delay channel diagram, to ensure that multimodal signals can dynamically adapt and maintain rhythmic coordination during the reconstructed time path, multiple control structures need to be sequentially activated based on the path planning of the delay channel diagram. This involves unified allocation of speech, image, and motion signals at the micro-time unit and beat organization levels, ensuring that the three types of signals maintain a stable and synchronized output sequence even under complex delay conditions. The specific steps are as follows: Based on the path segments corresponding to each signal type in the delay channel diagram, a micro-periodic misalignment choreography engine is activated to perform rhythmic correction and adjustment on the speech, image, and motion signals at a micro-timescale. In the delay channel diagram, some signals experience time misalignment due to detours or buffering, creating slight deviations in the original beat structure. While these deviations are insufficient to cause overall misalignment, they can still produce a sense of rhythmic incoordination at the perceptual level. Therefore, each signal sequence is divided into micro-periodic units, each containing a fixed number of trigger nodes, with the intervals between nodes controlled by the beat parameters provided by the channel diagram. In practice, the signal trigger timing within each micro-period is redistributed to match its beat center with the path rhythm in the channel diagram. Simultaneously, a slight misalignment is introduced as a beat tension adjustment mechanism. Through this tension creation and release process, adaptive balancing of the signal rhythm at the micro-level is achieved, thus correcting beat consistency without disrupting the overall rhythmic structure.

[0060] To address rhythmic gaps or overlaps that may occur in image signals during delayed paths due to frame rate changes or inter-frame interval adjustments, an image frame interpolation control mechanism is activated. This mechanism constructs a shadow frame interpolation structure to compensate for image content and facilitate rhythmic transitions between key time points. Specifically, for regions in the re-arrangement path where inter-frame distances exceed a threshold, a dynamically generated set of shadow frames is inserted between the original and target frames. These shadow frames do not completely reconstruct new content but instead construct transitional content by referencing visual elements such as motion trajectories, color changes, and structural boundaries of preceding and following frames. This fills the inter-frame gaps in the temporal dimension and maintains the continuity of the image content in the spatial dimension. Simultaneously, for frame overlap regions caused by inter-frame interval compression, interpolation is employed. This involves retaining the main frame signal and temporarily hiding the excess portion to prevent visual jumps or flickering. Through the shadow frame interpolation mechanism, the image signal maintains a stable visual rhythm output in both dense and sparse beat regions, ensuring a natural connection between the image and sound beats in conjunction with audio and motion signals.

[0061] To address the phase shift issue that occurs in multi-path speech signal propagation, a phase conjugate flow structure is activated to adjust and align the phase state of the speech waveform across different beat intervals. Because the speech signal may experience beat delays relative to the image or action path during its propagation, its waveform period may exhibit pre- or post-phase differences, causing a shift in the speech rhythm that should occur at the lip movements or action points in the image. To overcome this problem, a phase-aware node is established at the entry point of each beat to record the current phase start point of the speech signal, and a phase alignment node is established at the exit point of the beat to compare the deviation range of the speech waveform's trajectory within that beat with the ideal rhythmic track. When a significant shift is detected, a conjugate mechanism is activated, symmetrically reversing the shifted phase and shifting the start time of the speech waveform forward or backward to achieve a conjugate match with the current beat structure. This operation does not change the frequency or timbre of the speech content itself, but only adjusts its presentation window on the time axis, thus ensuring precise alignment between the speech, image, and action rhythm at the point of articulation.

[0062] After adjusting the rhythmic structure of the speech and image signals, to ensure that the action signal can also maintain a synchronous response within the same time structure, a speech breathing rhythm grid structure and an action hysteresis valve chain structure are activated to achieve fine control over the rhythmic response speed of the action signal. The core of the speech breathing rhythm grid structure lies in simulating the breathing gaps in natural speech, introducing micro-periodic pauses into the speech rhythm to create a rhythmic fluctuation with varying tension and relaxation. A rhythm grid structure is set at these rhythmic troughs, allowing the triggering mechanism of the action signal to switch states synchronously with the speech rhythm. Through this breathing-like rhythmic control, the action signal can obtain a predictable trigger window. The action hysteresis valve chain structure controls the action signal to not respond immediately after receiving the trigger command, but rather to release it with a short delay based on the convergence point of the speech rhythm and image rhythm. This delayed release structure consists of multi-level threshold control units. Each threshold is determined by the frame state of the image signal and the rhythmic position of the speech signal at that moment. When the conditions are met, the action signal is released, creating a natural hysteresis effect, thereby improving the overall coordination and rhythmic consistency of the multimodal actions. Through the combined effect of the above-mentioned multiple structures, speech signals, image signals, and motion signals can maintain a stable, coordinated, and rhythmic time output state in the reconstruction path under the guidance of the delay channel diagram, providing a high-quality rhythm control foundation for multimodal content generation.

[0063] This invention establishes a nanosecond-level time reference to ensure consistent acquisition of multi-source data at the microscopic time scale. By constructing temporal texture bands and energy progression sequences, it accurately identifies the temporal offset regions of multimodal signals, extracts pseudo-synchronization peak segments, and then embeds directional reverse time markers at key time points to form a traceable time anchor structure, achieving comprehensive perception and quantitative expression of signal drift paths. Through the rearrangement of the time flow, a delay channel map with buffering and detour capabilities is constructed, enabling multimodal signals to dynamically adapt to different paths, significantly enhancing the system's fault tolerance to complex temporal fluctuations. Finally, combined with a refined beat control mechanism and a multi-layered synchronization structure, it ensures that speech, image, and motion signals maintain a highly consistent output rhythm and coordinated expression throughout the long-term content generation process, significantly improving the stability and credibility of multimodal generated content in terms of perceptual coherence, semantic accuracy, and user experience.

[0064] This invention provides, for example Figure 2 The system shown is a new media multimodal content production system based on artificial intelligence, including a timing reference module, a misalignment detection module, a drift marking module, a channel arrangement module, and a synchronization control module. The timing reference module establishes a nanosecond-level timing reference by acquiring speech beat signals, image frame edge signals, and action key point signals to construct a temporal texture band in a continuous time dimension, which is used as a time alignment reference for subsequent multimodal signals. The misalignment detection module constructs an energy progression sequence of speech signals, image signals, and action signals based on the temporal distribution characteristics in the temporal texture band. It converts the arrival order of multimodal signals into energy ladders and identifies peak segments with temporal misalignment between the signals, generating pseudo-synchronous peak sequences. The drift marker module embeds reverse time markers at the leading and trailing edges of each pseudo-synchronization peak segment to record the direction and amplitude of time drift between signals, forming a set of reverse anchor points with time traceability capability. The channel orchestration module, based on the reverse anchor point set, re-orchestrates the time stream and generates a delay channel diagram containing buffers and detour areas by adjusting the trigger order and beat interval, giving the time stream multi-path redundancy capability. The synchronization control module, based on the structural relationship of the delay channel diagram, sequentially activates the micro-periodic misalignment choreography engine, shadow frame insertion mechanism, phase conjugate diversion structure, voice breathing rhythm grid structure, and motion hysteresis valve chain structure, so that the time series of voice signal, image signal, and motion signal maintain a synchronous and stable state.

[0065] This invention provides a method for producing new media multimodal content based on artificial intelligence, which is implemented through the aforementioned system for producing new media multimodal content based on artificial intelligence. For details on the specific methods and processes of the system for producing new media multimodal content based on artificial intelligence, please refer to the embodiment of the method for producing new media multimodal content based on artificial intelligence, which will not be repeated here.

[0066] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for producing new media multimodal content based on artificial intelligence, characterized in that, Includes the following steps: S001, establish a nanosecond-level time reference by collecting speech beat signals, image frame edge signals and action key point signals to construct a temporal texture band in the continuous time dimension for subsequent multimodal signal time alignment reference; S002, based on the temporal distribution characteristics in the temporal texture band, constructs the energy progression sequence of speech signals, image signals and action signals, transforms the arrival order of multimodal signals into energy ladders, and identifies peak segments with temporal misalignment between signals to generate pseudo-synchronous peaks; S003, embed reverse time markers at the leading and trailing edges of each pseudo-synchronization peak segment to record the time drift direction and amplitude between signals, forming a set of reverse anchor points; S004, based on the reverse anchor point set, rearranges the time stream, generates a delay channel diagram by adjusting the trigger order and beat interval, and gives the time stream multi-path redundancy capability; S005, based on the structural relationship of the delay channel diagram, sequentially activates the micro-periodic misalignment choreography engine, shadow frame insertion mechanism, phase conjugate diversion structure, voice breathing rhythm grid structure and motion hysteresis valve chain structure, so that the time series between voice signal, image signal and motion signal maintain a synchronous and stable state.

2. The method for producing new media multimodal content based on artificial intelligence according to claim 1, characterized in that, Step S001 includes: A unified time reference in nanoseconds is constructed using a high-frequency sampling clock, and the voice input device, image acquisition device and motion capture device are configured to synchronize clocks. A unified time trigger signal is used to ensure that the sampling start points of each device are strictly consistent in time. During the data acquisition process, the speech signal is sampled based on the beat of sound pressure fluctuation, the image signal is sampled based on the frame edge change, and the motion signal is sampled based on the extreme value change point of joint position. Each sampling point is assigned a unique timestamp and assigned to a unified time coordinate axis. Based on whether there are changes in speech rhythm, image frame edge, or key points of action at each time point, event markers are established and classified, and changes in signal response intensity and occurrence order are recorded to form a sequence of perceptual events. A complete temporal texture band is constructed based on the perceived event sequence. The high-frequency response area is extracted and summarized to form a temporal structure with temporal readability and continuous triggering pattern, which is used to guide the time alignment of subsequent multimodal signals.

3. The method for producing new media multimodal content based on artificial intelligence according to claim 2, characterized in that, During the establishment of event markers, a dynamic weight adjustment strategy is introduced for each type of signal. Based on the interleaving frequency of multimodal signals at the same time point, the expression intensity of the signal at the interleaving moment is enhanced, thereby improving the accuracy of the perception event sequence in depicting the time triggering relationship.

4. The method for producing new media multimodal content based on artificial intelligence according to claim 2, characterized in that, Step S002 includes: On the established temporal texture band, the frequency of occurrence, amplitude of change and continuous trigger span of speech signal, image signal and action signal in each time interval are extracted segment by segment to generate time feature points; Based on the arrangement of time feature points on the time axis, the energy change trends of each signal are sampled and superimposed to form a continuous energy ladder consisting of multiple energy increase segments and energy decay segments, thus constructing an energy progression sequence under a unified time reference. The peak time positions of speech signals, image signals and action signals in the energy progression sequence are compared, the peak time difference and energy amplitude difference are calibrated, the trigger misaligned segments are identified, and a peak index table is generated. Based on the peak index table, extract the time-offset peak groups that appear periodically or quasi-periodically, match the peak pairs to generate pseudo-synchronous peak columns, and establish a one-to-one correspondence with the time-series texture bands.

5. The method for producing new media multimodal content based on artificial intelligence according to claim 4, characterized in that, When constructing an energy progression sequence under a unified time reference, a smoothing process is used to transition and adjust the energy abrupt change regions between adjacent energy segments, so that the energy changes of speech signals, image signals and action signals on the time axis form a continuous and stable trapezoidal distribution.

6. The method for producing new media multimodal content based on artificial intelligence according to claim 4, characterized in that, Step S003 includes: Based on the energy progression sequence, the leading edge and trailing edge boundaries of pseudo-synchronous peak segments are determined one by one. By analyzing the changes in the energy curves of speech signals, image signals, and action signals, the time points at which the signals begin to deviate and realign are extracted as segment boundaries. Time markers with direction information and time interval parameters are embedded at each leading and trailing edge boundary. The drift direction and time offset are recorded by comparing the peak signal energy with the standard reference position. Each time marker is bound to its corresponding timeline position to build a set of anchor points with time as the main index, and a chain connection is established according to the order of peak segments to form a complete time traceability structure. By jointly mapping the anchor point set with the energy progression sequence and pseudo-synchronous peaks, a multi-layered data structure containing drift direction, energy trend, and temporal position is established to support subsequent time re-arrangement and synchronization control.

7. The method for producing new media multimodal content based on artificial intelligence according to claim 6, characterized in that, Step S004 includes: Based on the set of reverse anchor points, the offset trend and distribution density of the signal on the time axis are extracted, and a set of time sequence segments is constructed as the basic framework for rearrangement. Based on the set of time segments, the triggering order and beat interval of the signals in each segment are adjusted to maintain the continuity and consistency of the rhythmic relationship between the signals and to construct a rhythmically coordinated local time flow structure. The locally reconstructed paragraphs are globally stitched together, and buffer and bypass areas are inserted between the paragraphs to construct a delay channel diagram; Based on the structural relationship of the delay channel diagram, the path number is bound to the signal triggering command to establish a dynamic channel mapping, which enables the time flow to have multi-path redundancy and rhythm stability.

8. A method for producing new media multimodal content based on artificial intelligence according to claim 7, characterized in that, The buffer area is used to absorb the time error caused by the trigger beat adjustment. The duration is dynamically set according to the drift difference between adjacent segments. The detour area is used to build a delay branch for the signal. After the adjustment is completed through an independent channel outside the main path, it returns to the main channel.

9. A method for producing new media multimodal content based on artificial intelligence according to claim 7, characterized in that, Step S005 includes: Based on the signal path segments in the delay channel diagram, the micro-periodic misalignment choreography engine is activated to correct and adjust the micro-time rhythm of the voice signal, image signal, and motion signal. To address rhythmic gaps and overlapping areas in image signals during frame rate changes, a shadow frame interpolation mechanism is activated to perform image transition compensation and frame content coordination. To address the phase shift of the speech signal in the delay path, a phase conjugate diversion structure is activated to adjust the start time of the speech waveform to achieve beat alignment. By using a voice breathing rhythm grid structure and an action hysteresis valve chain structure, the rhythm synchronization and delayed release of action signals are achieved, so that each signal maintains a stable output rhythm relationship.

10. A system for producing new media multimodal content based on artificial intelligence, used to implement the method for producing new media multimodal content based on artificial intelligence as described in any one of claims 1-9, characterized in that, It includes a timing reference module, a misalignment detection module, a drift marker module, a channel orchestration module, and a synchronization control module: The timing reference module establishes a nanosecond-level timing reference by acquiring speech beat signals, image frame edge signals, and action key point signals to construct a temporal texture band in a continuous time dimension, which is used as a time alignment reference for subsequent multimodal signals. The misalignment detection module constructs an energy progression sequence of speech signals, image signals, and action signals based on the temporal distribution characteristics in the temporal texture band. It converts the arrival order of multimodal signals into energy ladders and identifies peak segments with temporal misalignment between the signals, generating pseudo-synchronous peak sequences. The drift marker module embeds reverse time markers at the leading and trailing edges of each pseudo-synchronization peak segment to record the time drift direction and amplitude between signals, forming a set of reverse anchor points. The channel orchestration module, based on the reverse anchor point set, re-orchestrates the time stream, generates a delay channel diagram by adjusting the trigger order and beat interval, and gives the time stream multi-path redundancy capability. The synchronization control module, based on the structural relationship of the delay channel diagram, sequentially activates the micro-periodic misalignment choreography engine, shadow frame insertion mechanism, phase conjugate diversion structure, voice breathing rhythm grid structure, and motion hysteresis valve chain structure, so that the time series of voice signal, image signal, and motion signal maintain a synchronous and stable state.

Images