A generative multimedia content authoring method

By collecting implicit behavioral data to generate spatiotemporal satisfaction heatmaps and data generatrices, the parameters in the generative multimedia creation method are automatically adjusted, solving the problems of delayed user interaction feedback and low efficiency of multi-user collaborative creation, and realizing efficient and personalized multimedia content generation.

CN122640608APending Publication Date: 2026-08-25BEIJING HUAYI ZHONGXIN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610878797.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-06-17
Publication Date
2026-08-25

AI Technical Summary

Technical Problem

Existing generative multimedia creation methods have shortcomings in terms of real-time user interaction feedback, precise control of generation parameters, naturalness of content transitions, and efficiency of multi-user collaborative creation. In particular, the lack of effective modeling and utilization of implicit behaviors limits the personalization and adaptability of generated content.

Method used

By collecting implicit behavioral data to generate spatiotemporal satisfaction heatmaps, constructing data genealogy diagrams, accurately locating problem areas and upstream nodes, automatically adjusting parameters, achieving adaptive creation, supporting multi-user collaborative editing, and handling complex conflicts.

Benefits of technology

It improves the directionality and precision of parameter adjustment, realizes pixel-level continuous parameter changes and transition effects, and enhances the efficiency of multi-user collaborative creation and the personalization of generated content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122640608A_ABST
    Figure CN122640608A_ABST
Patent Text Reader

Abstract

The application discloses a generative multimedia content creation method, comprising: generating a space-time satisfaction heat map by collecting implicit behavior data of a user, reflecting the implicit satisfaction degree of the user to multimedia content in real time, combining a workflow to build a data pedigree chart, extracting negative value voxels from the heat map to form a problem area, and tracing and positioning an upstream target node generating the problem along a reverse edge of the workflow, then extracting parameter types from the target node, automatically adjusting adjustable parameters according to heat map feedback characteristics and adjustment rules, triggering new output by re-executing generation, forming a new round of feedback through comparison between new and old outputs, and storing the adjusted parameters into a user portrait database, so that the same type of node in a newly created workflow can be directly reused to realize personalized adaptive adjustment. The method does not need active operation of the user, reduces feedback lag through implicit behavior collection, accurately locates a problem source by using the heat map and the pedigree chart, and improves directionality and accuracy of parameter adjustment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of content generation technology, and in particular to a generative multimedia content creation method. Background Technology

[0002] With the rapid development of AI-generated content (AIGC) technology, generative multimedia content creation has become an important development direction for the digital content industry. Currently, deep learning-based generative models (such as diffusion models, generative adversarial networks, and variational autoencoders) are widely used in the automatic generation and editing of multimedia content such as images, videos, audio, and text. However, existing generative multimedia creation methods still face many technical bottlenecks in practical applications, especially in terms of real-time user interaction feedback, precise control of generation parameters, natural content transitions, and efficiency of multi-user collaborative creation, where systematic solutions are still lacking.

[0003] First, existing generative creation systems largely rely on explicit user feedback to adjust their generation strategies, such as requiring users to manually rate, select preferences, or modify parameters multiple times. This explicit feedback mechanism not only increases the cognitive burden on users but also suffers from significant feedback lag, failing to capture user satisfaction levels in real time during the creation process. Simultaneously, the lack of effective modeling and utilization of implicit behaviors (such as gaze duration, playback behavior, and rapid swiping) makes it difficult for the system to proactively perceive user intent, limiting the personalization and adaptability of generated content.

[0004] Secondly, current workflow-driven generation systems typically employ fixed parameter configurations, lacking the ability to control the gradual changes in parameters across time and space. For example, in video generation, users might desire a lower stylization intensity in the first half and a gradual increase in the second half; in image generation, they might prefer a cooler tone on the left and a warmer tone on the right. However, existing methods often use globally uniform parameters, failing to achieve continuous parameter changes at the pixel or frame level. Furthermore, transition effects are usually applied uniformly across the entire image, ignoring the differences in visual features across different semantic regions. This can lead to visual jumps or information loss during transitions, affecting the coherence of the content and the viewing experience.

[0005] Furthermore, with the increasing demand for team collaboration, generative multimedia authoring tools urgently need to support real-time collaborative editing by multiple users. However, most existing systems employ serial locking mechanisms or manual merging methods, requiring users to wait for others to complete their operations before they can edit, severely reducing team creation efficiency. Even with concurrent operation support, existing solutions still have significant shortcomings in conflict detection, operation transformation, and semantic merging. They struggle to handle complex conflict scenarios such as concurrent deletion of nodes and modification of parameters, and multiple connections to the same source port. They lack automated solutions for issues like circular dependencies, parameter overriding, and connection conflicts, and cannot achieve intelligent fusion and personalized recommendations for conflicting content. Summary of the Invention

[0006] This application provides a generative multimedia content creation method that obtains user feedback in real time through implicit behavior collection, reducing feedback lag issues. Based on spatiotemporal satisfaction heatmaps and data generatrices, it accurately locates problem areas and upstream nodes, improving the directionality and accuracy of parameter adjustments. It constructs an adaptive creation method that automatically collects and utilizes implicit behavior feedback to precisely drive the fine-tuning of upstream node parameters without increasing the burden of active user operation.

[0007] This application provides a generative multimedia content creation method, including: S101, collect implicit behavioral data, and generate a spatiotemporal satisfaction heatmap based on the implicit behavioral data through a matrix; wherein, the implicit behavioral data includes video operation behavior data, image operation behavior data, audio operation behavior data, and text operation behavior data; S102, obtain the workflow of multimedia real-time operation, construct a data genealogy diagram based on the workflow, extract negative value voxels from the spatiotemporal satisfaction heatmap to form a problem region set, trace back along the reverse edge of the workflow according to each voxel in the problem region set to obtain the upstream node set, and identify the upstream target node that generated the problem region from the upstream node set. S103: Extract parameter types from upstream target nodes, obtain heatmap feedback features from the problem area of ​​the spatiotemporal satisfaction heatmap, and adjust adjustable parameters according to parameter types and heatmap feedback features using adjustment rules; S104, triggers re-execution based on the adjusted parameters, re-executes content creation to generate new output, compares the new output with the original output, and forms new feedback based on the comparison results; constructs a user profile database based on the adjusted parameters, and if a node of the same type is added to a new workflow, directly uses the data in the user profile database.

[0008] Preferably, the collection of implicit behavioral data includes: setting interactive event listeners in the video, image, audio, and text operation areas; setting a stillness threshold and a dwell threshold; recording a positive interest signal when the user's gaze exceeds the stillness threshold; recording a weak negative signal when the user quickly swipes over and stays for less than the dwell threshold; and classifying a segment as a positive satisfaction signal or a negative dissatisfaction signal based on whether an editing operation is performed after playback when the same segment is replayed twice or more.

[0009] Preferably, the method for generating the spatiotemporal satisfaction heatmap is as follows: constructing a heatmap based on video operation behavior data. × × A three-dimensional matrix, in which The total number of frames in the video. For spatial resolution, construct image manipulation behavior data × Two-dimensional matrix To achieve spatial resolution, a one-dimensional matrix of length N is constructed for audio operation behavior data, and a one-dimensional matrix of length L is constructed for text operation behavior data. All matrices are initialized to 0, and their values ​​are limited to the range [-1, 1]. Negative values ​​represent implicit user dissatisfaction, while positive values ​​represent implicit satisfaction or interest.

[0010] Preferably, the steps for generating the upstream target node of the problem area are as follows: adjacent negative voxels in the heatmap that are less than a preset threshold are clustered into problem areas using a connected component labeling algorithm. Starting from the currently reviewed output node, the time index is matched according to the modality type of the problem area, and the process is traced back along the reverse edge of the workflow. The maximum tracing depth is set to 10 layers, and each node is associated with a cumulative contribution weight. The node with the highest weight is taken as the upstream target node.

[0011] Preferably, the content creation method further includes: S201, normalize the spatiotemporal satisfaction heatmap to generate a weight map, generate anchor points by setting the extreme values ​​of parameters, and weight and fuse each pixel with different anchor points according to the weight map to obtain the parameter gradient output content; S202: Divide the created content into regions to obtain several regions, configure a transition strategy for each region, perform asynchronous transitions between regions based on the transition strategy, and splice them together to obtain the spliced ​​content; wherein, the transition strategy includes transition type, transition duration, and transition curve; S203, input the output content of parameter gradient and the spliced ​​content into step S101 for review, and execute steps S101-S104.

[0012] Preferably, the transition types include four types: fade, wipe, zoom, and dissolve; the transition duration for different regions is set independently, in seconds; the transition curve controls the rate of change during the transition process, including linear constant speed, slow in, slow out, and slow in slow out; in asynchronous transition rendering, the transition processes of different regions occur asynchronously on the timeline, and in the same frame, some regions have completed the transition, some regions are still transitioning, and some regions have not yet started transitioning.

[0013] Preferably, the content creation method further includes: S301: Obtain editing operation data from multiple users, and construct a multi-branch workflow diagram based on the editing operation data from multiple users using operation transformation; S302 uses detection rules to identify conflict types based on the editing operation data of multiple users, merges the identified conflict types, and forms a unified workflow state; S303, detect the merging results of conflict types. If merging is not possible, trace the workflow status using playback and rollback strategies based on the editing operation data of multiple users; and perform interactive design on the content creation interface.

[0014] Preferably, the operation transformation includes: when deleting a node and modifying a parameter concurrently, retaining the deletion operation and converting the modification into an invalid operation; when different connections on the same source port are concurrently used, automatically creating a multi-path merge node; when adding a node and deleting a node concurrently, prioritizing deletion unless it is a temporary system node; when moving a node position and deleting a node concurrently, retaining the deletion and moving operations as invalid; when modifying the same parameter concurrently, converting it into a combined operation carrying two candidate values ​​for semantic merging processing.

[0015] Preferably, the semantic merging method is as follows: For two output contents that need to be merged due to parameter coverage conflicts or connection conflicts, their respective low-dimensional feature vectors are extracted—the color histogram and texture spectrum of the image, the Mel spectrum and rhythm features of the audio, and the optical flow motion features of the video. Then, the similarity score between the two feature vectors is calculated using a similarity formula. Based on the similarity level, a fusion method is selected. If the similarity is higher than 0.8, weighted stacking is selected; if the similarity is between 0.4 and 0.8, content masking is selected, that is, the image is divided into foreground / background through semantic segmentation, which are taken from different sources; if the similarity is lower than 0.4, time stitching or picture-in-picture is recommended, and fusion is not forced.

[0016] Preferably, the replay and rollback strategy when the conflict cannot be merged is as follows: Edit operation data containing a global sequence number, parent operation sequence number, operation type, user ID, timestamp, and workflow snapshot hash is read from the server; the user replays each operation sequentially using the timeline replay function; the minimum change set is calculated using an inverse operation transformation algorithm; during rollback, two options are provided: forced rollback discarding all subsequent modifications or partial rollback rolling back only the directly affected parts; private branches can be created from any historical operation, edited independently, and then merged back into the main branch via a merge request; during merging, conflict detection and semantic merging are triggered again.

[0017] One or more technical solutions provided in this application have at least the following technical effects or advantages: by implicit behavior collection, user feedback is obtained in real time, reducing the problem of feedback lag; based on spatiotemporal satisfaction heatmaps and data genealogy diagrams, problem areas and upstream nodes are accurately located, improving the directionality and accuracy of parameter adjustment; and an adaptive creation method is constructed that automatically collects and utilizes implicit behavior feedback to precisely drive the fine-tuning of upstream node parameters without increasing the burden of active user operation.

[0018] By using superimposed differential generation and weighted graphs to control the fusion ratio of different anchor point versions, it achieves parameter gradient effects such as low cartoon intensity in the first half of the video and high intensity in the second half, or warm tones on the left and cool tones on the right of the image. The asynchronous regional transition divides the screen into multiple regions, and each region independently controls the transition type and rate, solving the problem of parameter space variation and the mismatch between transition effects and visual features in generative multimedia content creation, and achieving pixel-level continuous parameter changes and transition effects. By transforming graph structures through operations, detecting conflicts, and semantic merging, conflict types are resolved. Unmergable conflict types are resolved through replay and rollback strategies, reducing user effort and enabling multiple users to drag nodes, connect lines, and modify parameters simultaneously without waiting for lock release. Attached Figure Description

[0019] Figure 1 This is a block diagram of a generative multimedia content creation method according to the present invention; Figure 2 This is a flowchart illustrating the output content of the parameter gradual change in this invention; Figure 3 This is a schematic diagram illustrating the process of constructing a multi-branch workflow diagram for this invention. Detailed Implementation

[0020] To facilitate understanding of the present invention, a more complete description of this application will be given below with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to enable a more thorough and complete understanding of the disclosure of the present invention.

[0021] It should be noted that the terms "vertical," "horizontal," "up," "down," "left," "right," and similar expressions used in this article are for illustrative purposes only and do not represent the only possible implementation.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0023] Example 1: Figure 1 This is a flowchart illustrating a generative multimedia content creation method according to an embodiment of the present invention, including: S101, collect implicit behavioral data, and generate a spatiotemporal satisfaction heatmap based on the implicit behavioral data through a matrix; wherein, the implicit behavioral data includes video operation behavior data, image operation behavior data, audio operation behavior data, and text operation behavior data; Specifically, interactive event listeners are set up in the video player, image viewer, audio waveform editor, and text display area to collect implicit behavioral data. Specifically, a stillness threshold and a dwell threshold are set. In the video playback area, when the mouse pointer remains stationary on the video screen for longer than the preset stillness threshold, it is determined that the user is looking at the corresponding area of ​​the screen, and the time of the current video frame and the normalized coordinates of the mouse in the screen are recorded, and this event is marked as a positive interest signal. When the mouse quickly moves across a certain area and stays in that area for less than the preset dwell threshold before moving away, it is determined that the user is ignoring the content of that area, which is considered a weak negative signal, and recorded. The system records the starting and ending coordinates of the traced path and calculates the time window and spatial range covered by the path. For the timeline, the system monitors dragging and playback behavior. When a user repeatedly drags and replays the same segment two or more times, and the playback speed does not exceed normal playback speed, it indicates that the user is highly attentive to the content of that time period. If no zooming, drawing, or editing operations are performed after playback, it is considered a positive satisfaction signal. If the image is zoomed or drawn immediately after playback, it is considered a negative dissatisfaction signal. The start and end times of the segment are recorded. For image operation behavior data collection, the system collects two types of behavior in the image viewer: mouse wheel zooming and mouse movement. When the user uses the mouse... When the user zooms in using the scroll wheel or by clicking the zoom button, it indicates a need to examine the details of the zoomed-in area. The normalized coordinates of the zoom center point and the zoom level are recorded. If the zoomed-in period exceeds 2 seconds without any subsequent modifications, it is considered a positive signal; if the zoomed-in period immediately ends and the user jumps to another area, it is considered a negative signal. The mouse remaining stationary on the image for more than a preset static threshold is considered a gaze, and its coordinates are recorded; a rapid mouse hover is considered a negative dissatisfaction signal. For audio operation behavior acquisition, in the audio waveform editor, in addition to volume adjustment in the video playback area, dragging and positioning on the timeline and looping of segments are also monitored. When the user repeatedly drags to the same time interval and replays it more than twice, and the replay... If the waveform is not clipped or marked after playback, it is considered a positive satisfactory signal; if clipping or muting is performed immediately after playback, it is considered a negative signal. All audio operation events are recorded within a time window (start and end time offset). For text operation behavior acquisition, in the text display area, the system listens for mouse selection of words, right-click menus, and the keyboard delete key. When a user selects a piece of text, if they select "regenerate" via the right-click menu or directly press the delete key, it is considered a strong negative signal for that piece of text. The system records the starting and ending character indices of the selected text. If the user only selects words without any subsequent operation, it is only observed as neutral and no signal is generated. The implicit behavior data collected above is stored.

[0024] A spatiotemporal satisfaction heatmap is generated based on the aforementioned collected implicit behavioral data. This heatmap is essentially a high-dimensional matrix; for video operation behavior data, a three-dimensional matrix with a size of [missing information]. × × ,in The total number of frames in the video. To achieve spatial resolution, each voxel in the 3D matrix is ​​initialized to 0, with values ​​limited to [-1, 1]. Negative values ​​represent implicit user dissatisfaction, while positive values ​​represent implicit satisfaction or interest. For image operation behavior data, a 2D matrix with size [missing information] is constructed. × Each pixel is initialized to 0; for audio operation data, a one-dimensional matrix of length is constructed. ,in The total number of audio sampling points is given, with each time unit initialized to 0. For text operation behavior data, a one-dimensional matrix of length L is constructed, where L is the total number of characters in the text, and each word position is initialized to 0.

[0025] S102, obtain the workflow of multimedia real-time operation, construct a data genealogy diagram based on the workflow, extract negative value voxels from the spatiotemporal satisfaction heatmap to form a problem region set, trace back along the reverse edge of the workflow according to each voxel in the problem region set to obtain the upstream node set, and identify the upstream target node that generated the problem region from the upstream node set. Furthermore, during workflow execution, the source of each spatiotemporal unit of each output content is recorded—that is, which upstream node's output field and what transformation it underwent to generate the unit. This record structure is called a data genealogy graph. The genealogy graph is essentially a directed acyclic graph (DAG), where nodes represent production or input nodes in the workflow, and edges represent data dependencies. Whenever a production node completes execution and generates output, that node generates a data genealogy graph. All voxels in the spatiotemporal satisfaction heatmap are traversed, and voxels with values ​​less than a preset threshold are added to the candidate set. Since negative signals from adjacent voxels often originate from the same problem region, adjacent negative voxels are clustered into connected components, forming several independent problem regions, i.e., a problem region set. Each problem region is represented by its smallest bounding box, and a three-dimensional connected component labeling algorithm is used to label all adjacent negative voxels as the same problem region. For each extracted problem region, the records in the genealogy data table are queried. The process is as follows: The query starts with the output node currently being reviewed by the user. This node can be the final output node of the workflow or an intermediate node selected by the user through the interface. During the query, the corresponding fields are matched according to the modality type of the problem area: for video or audio, the output time index in the genealogy record must fall within the time range defined by the problem area; for images or videos, it is also necessary to check whether the spatial coordinates are inside the spatial bounding box of the problem area; for text, it is checked whether the character position index is within the character range of the problem area. From the query results, the node ID of each record is extracted to form the first-level upstream node set. The total weight of each upstream node's contribution to the problem area is recorded. The process continues to trace along the reverse edge of the workflow until the input node or the original generation node is reached. For each node in the first-level upstream node set, the genealogy record of that node is queried to find its upstream node, which is then added to the second-level upstream node set, and so on. To avoid infinite recursion, the maximum tracing depth is set to 10 layers. Finally, all reachable upstream nodes starting from the problem area are obtained. Each node is accompanied by a cumulative contribution weight, which is the sum of the products of the contribution weights on all paths from the problem area to the node. All upstream nodes are sorted from high to low according to their cumulative contribution weights, and the node with the highest weight is taken as the upstream target node.

[0026] S103: Extract parameter types from upstream target nodes, obtain heatmap feedback features from the problem area of ​​the spatiotemporal satisfaction heatmap, and adjust adjustable parameters according to parameter types and heatmap feedback features using adjustment rules; Specifically, the node metadata table is queried using the identifier of the upstream node to obtain all adjustable parameters of the upstream node. Heatmap feedback features are obtained from the problem areas of the spatiotemporal satisfaction heatmap. These features include the concentration of negative values ​​within the problem area and the distribution of the severity of negative values. For the concentration of negative values ​​within the problem area, a ratio threshold is set by calculating the average heatmap value of all voxels in the problem area and the proportion of voxels in that area to the total number of voxels in the output content. If the calculated ratio is greater than the preset ratio threshold, it indicates a widespread distribution of dissatisfaction; if the calculated ratio is less than the preset ratio threshold, it indicates that the problem is concentrated locally. For the distribution of the severity of negative values ​​within the problem area, the variance and maximum value of negative values ​​within the area are calculated. If the variance is greater than the variance threshold, it indicates uneven dissatisfaction and the possible existence of multiple sub-problems; the maximum value indicates that the area is strongly disliked by users. Based on the obtained heatmap feedback features and parameter types, the adjustable parameters are adjusted. The adjustment steps are as follows: when the parameter type is intensity or ratio... When the problem area heatmap shows a concentration of negative values ​​and the negative values ​​decrease, the adjustment parameter's magnitude (step size) is increased. If the negative values ​​in the problem area are more severe than before, the adjustment is reversed, and the step size is halved. When the parameter type is a discrete option (sampling method, filter type), an option list is set up. The option list refers to arranging all the values ​​of the discrete parameter into an ordered list. Each adjustment moves one position to the right along the option sequence and takes the next value. If the end of the sequence has been reached, the loop returns to the first option. If the negative values ​​of the heatmap do not improve after two consecutive adjustments, the next adjustment skips that option and moves directly to the next option in the sequence. When the parameter type is text-based, keywords are extracted from the user's implicit behavior. For example, if the user repeatedly highlights words to replace cartoon with realistic, the prompt word "cartoon" is automatically replaced with "realistic". For the parameter type is a spatiotemporal range, the parameter is fine-tuned according to the time boundary of the problem area to avoid negative feedback time points. For example, if the user frequently skips the 3rd to 5th second, and there is a video cropping parameter of [0, 10] seconds, the cropping starting point is automatically changed to 5 seconds.

[0027] S104, triggers re-execution based on the adjusted parameters, re-executes content creation to generate new output, compares the new output with the original output, and forms new feedback based on the comparison results; constructs a user profile database based on the adjusted parameters, and if a node of the same type is added to a new workflow, directly uses the data in the user profile database.

[0028] Furthermore, based on the adjusted parameters, the calculation is recalculated downstream from the adjusted node (upstream results are reused) to generate new outputs. The new outputs are compared with the original outputs, and new feedback is generated based on the comparison results. All adjustment records (parameter changes + corresponding heatmap changes) are stored in the user profile database. When a user creates a new workflow and adds nodes of the same type, the data in the user profile database is used directly.

[0029] The technical solutions in the above embodiments of this application have at least the following technical effects or advantages: by implicit behavior collection, user feedback is obtained in real time, reducing the problem of feedback lag; based on the spatiotemporal satisfaction heatmap and data spectrum diagram, the problem area and upstream node are accurately located, improving the directionality and accuracy of parameter adjustment; and an adaptive creation method is constructed that automatically collects and utilizes implicit behavior feedback to precisely drive the fine-tuning of upstream node parameters without increasing the user's active operation burden.

[0030] Example 2: Example 1 can adjust the globally constant parameters of nodes through implicit behavioral feedback, but it cannot achieve continuous spatial / temporal gradation of parameters. Traditional transitions treat the entire screen uniformly. When there are spatial differences between preceding and following segments, a uniform transition will disrupt the visual continuity of their respective areas. This example achieves pixel-level continuous parameter changes and transition effects by generating multiple anchor point versions and using programmable region fusion. Figure 2 As shown.

[0031] S201, normalize the spatiotemporal satisfaction heatmap to generate a weight map, generate anchor points by setting the extreme values ​​of parameters, and weight and fuse each pixel with different anchor points according to the weight map to obtain the parameter gradient output content; Furthermore, the steps for normalizing the spatiotemporal satisfaction heatmap to generate a weighted map are as follows: First, linearly map the values ​​of all voxels in the heatmap to the range of 0 to 1—the region with the lowest satisfaction (the most negative value) is mapped to 0, the region with the highest satisfaction (the most positive value) is mapped to 1, and intermediate values ​​are proportionally mapped. Then, an inversion process is performed: the larger the negative value in the original spatiotemporal satisfaction heatmap, the higher the weight value (indicating a greater desire for parameter change), while the lower the weight value in the positive value region (indicating a desire for parameters to remain unchanged). After normalization and inversion, the weighted map is obtained, where the value of each pixel is between 0 and 1, where 0 represents no parameter change and 1 represents maximum parameter change. Adjustable parameters are set... Two extreme values ​​are set, including a minimum value and a maximum value. The minimum value corresponds to the position with a weight of zero in the weight map, indicating that the parameter effect is expected to be the weakest in that region. The maximum value corresponds to the position with a weight of one in the weight map, indicating that the parameter effect is expected to be the strongest in that region. After setting, the system automatically calls the same production node twice: the first time, the parameter is fixed at the minimum value to generate a complete anchor point output (denoted as the low parameter version); the second time, the parameter is fixed at the maximum value to generate another complete anchor point output (denoted as the high parameter version). Based on the low parameter anchor point output, the high parameter anchor point output, and the weight map, all pixels are weighted and fused. For each pixel in the output content, the value at the corresponding position on the weight map is read to calculate the final value. The formula is: ,in, This indicates the numerical value assigned to the pixel located at the coordinate position in the output content. This represents the pixel value at the same spatial coordinates in the full anchor point version generated using the minimum value. This indicates that the pixel value at the same spatial coordinates in the generated full anchor point version is used. This represents the weight value of the spatial coordinates at frame t in the video. These represent the horizontal and vertical positions in spatial coordinates, respectively, with t being the time index. The output content of the parameter gradient is obtained through the above formula.

[0032] S202, the created content is divided into regions to obtain several regions, a transition strategy is configured for each region, and asynchronous transition of regions is performed based on the transition strategy and spliced ​​together to obtain the spliced ​​content; Specifically, a weighted graph is used as the basis for content segmentation. The content is divided into several semantically meaningful regions based on the weighted graph. The specific segmentation method is as follows: multiple independent connected regions are obtained from the weighted graph using a connected component extraction algorithm (e.g., the watershed algorithm). Each region corresponds to a relatively uniform semantic block in the image, such as different human figures, vehicles, or buildings. The position of the segmentation curve is adjusted by dragging control points. Each frame is divided into several non-overlapping regions, resulting in several regions. For each segmented region, a transition strategy is specified at the splicing point according to the creative intent. Specifically, the transition strategy includes transition type, transition duration, and transition curve. For the transition type, the user selects from preset transition effects, including fade (…). The transitions include: gradually becoming transparent before revealing new segments, wiping (gradually pushing away the old image from one side), scaling (zooming in or out), and dissolving (the old image slowly merging into the new image). For transition duration, each area has a different duration, measured in seconds. For example, a 0.2-second fast hard cut is set for the foreground character, while a 1-second slow fade is set for the background scene. The transition curve controls the rate of change during the transition, such as linear constant speed, gradual in (slow at the beginning, then accelerating), gradual out (fast at the beginning, then slowing down), and gradual in / gradual out. Based on the transition strategy configured above, asynchronous transition rendering is performed, meaning the transition processes of different areas do not occur synchronously on the timeline. The output content of the parameter gradient is divided into two areas, designated A and B. The transitions between areas A and B are then synchronized at specific time points. For each region, a transition is generated according to its configured transition type and duration. The initial transition frame sequence, for example, if the transition duration for the foreground region is 0.2 seconds, then the system will... arrive Between +0.2 seconds, each pixel within that region is calculated according to the hard-cutting rule: At any given moment, the pixel originates entirely from region A; At time +0.2, the pixels originate entirely from region B; at intermediate times, a rapid transition is achieved through a step function (or extremely short linear interpolation), while the background region transitions for 1 second. arrive Between +1 seconds, the pixels in this region are blended frame by frame according to the fade rule: the pixel value at time t = the pixel value of region A × (1 - ) +Pixel value of region B × During the rendering process, the system generates a complete image for each frame. Specifically, it iterates through all regions, calculating the transition pixel value for each pixel based on its region and time point. Since the transition duration varies across regions, some regions within the same frame may have completed the transition (Region B), while others are still transitioning (mixed A and B), and still others haven't started transitioning (Region A). The results from all these regions are merged in the same frame, forming a sequence of transition frames with asynchronous visual effects. This sequence is then stitched together chronologically with the pre-transition portion of Region A and the post-transition portion of Region B to create a complete video file (the stitched content). In this final video, the transition effects of different semantic regions present asynchronous and personalized visual characteristics, preserving the rapid switching of foreground objects while ensuring the slow changes in the background don't distract the viewer. The boundaries are smooth and the overall transition is fluid and natural. Users can directly use this result for subsequent creation or output.

[0033] S203, input the output content of parameter gradient and the spliced ​​content into step S101 for review, and execute steps S101-S104.

[0034] The obtained parameter gradient output and the spliced ​​content are input into step S101 for review, and steps S101-S104 are executed to allow the parameters to smoothly change with spatial position (from left to right) or time (from beginning to end).

[0035] The technical solutions in the above embodiments of this application have at least the following technical effects or advantages: by superimposed differential generation, the fusion ratio of different anchor point versions is controlled by weight map, so as to achieve parameter gradient effects such as low cartoon intensity in the first half of the video and high intensity in the second half, or warm tone on the left and cool tone on the right of the image. The asynchronous transition of regions divides the screen into multiple regions, and each region independently controls the transition type and rate, so as to solve the problem of parameter space change and the problem of mismatch between transition effect and visual features in generative multimedia content creation, and achieve pixel-level continuous parameter change and transition effect.

[0036] Example 3: The generative multimedia creation in Examples 1 and 2 above is single-user, single-threaded. When multiple creators need to edit the same workflow simultaneously, the existing models only have two options: either wait sequentially, with one person finishing before the next, or manually export and import segments and then merge them. Both methods severely disrupt the continuity of team creation. This example achieves team creation by constructing a multi-user real-time collaborative workflow, improving creation efficiency. Figure 3 As shown.

[0037] S301: Obtain editing operation data from multiple users, and construct a multi-branch workflow diagram based on the editing operation data from multiple users using operation transformation; Specifically, the method for obtaining editing operation data from multiple users is as follows: Monitor the interactive behavior of each user's editing client (browser and desktop application). This interactive behavior includes: adding nodes, deleting nodes, modifying node parameters, adding connections, deleting connections, and moving node positions (affecting only the layout, not the execution semantics). Whenever a user performs any of these operations, the client immediately encapsulates the operation into an atomic operation object. This atomic operation object contains an operation identifier, operation type, the ID of the target node, operation payload data, the user identifier executing the operation, and a timestamp. Each client sets up two buffers: a local operation buffer and a remote operation buffer. The local operation buffer is used to temporarily store data that the user has just generated but... Operations not yet submitted to the server; the remote operation buffer is used to temporarily store operations received from other users from the server, using version vectors to capture causal dependencies between operations; when the server receives a set of operations from different clients (or a client pulls remote operations from the server), it first performs causal sorting based on the version vectors, specifically: all operations are topologically sorted according to the partial order of their version vectors (i.e., the causal dependency graph). If two operations do not have a causal dependency (i.e., concurrent operations), the sorted operation sequence is sent to all clients. For concurrent operations (i.e., two operations without causal dependency), operation transformation is performed according to the operation type and target object, that is, the operands of one of the operations are modified so that regardless of the order in which they are executed... Okay, the final state is consistent and the semantics are correct. Based on the workflow diagram, set the operation transformation rules: For concurrent deletion and parameter modification of nodes: If user A deletes node X, and user B concurrently modifies the parameters of node X, after transformation: the deletion operation is retained, the parameter modification operation is converted to an invalid operation (i.e., it has no effect), and a notification pops up on user B's interface notifying them that the node being modified has been deleted by A and the modification has been ignored; For concurrent connections of different origin ports: If user A connects the output port of node X to node Y, and user B connects the same output port of node X to node Z, after transformation: both connections are retained, but the system automatically creates a multi-way merging node M, using both Y and Z as input nodes for M, and simultaneously changes the original connection to X→M. The outputs from M are sent to Y and Z respectively. The default merging strategy for the merged nodes depends on the output types of Y and Z: if both are video clips, timeline stitching is used; if both are images, weighted overlay or picture-in-picture is used. Users can manually modify the merging strategy later. For concurrent node addition and deletion, if user A adds node N and user B deletes node N, after transformation: deletion is usually prioritized because deletion often means that the user explicitly does not want the node to exist. However, it needs to be considered in context: if node N is a temporary node automatically generated by the system, the addition is retained; otherwise, the addition operation is converted into an invalid operation. For concurrent node movement and deletion, the movement does not affect the execution semantics, so after transformation, deletion is retained, and movement is invalid.When two users modify the same parameter concurrently (e.g., A sets it to 0.7 and B sets it to 0.3), the system cannot automatically determine the priority. In this case, the transformation rule is to convert the two modification operations into a single combined operation carrying two candidate values. The downstream merging step will treat this as a parameter overwrite conflict and handle it through semantic merging.

[0038] When each user first opens a workflow, a snapshot of the current workflow (i.e., a serialized representation of all nodes and connections) is loaded from the server. This snapshot can be considered an initial version with all version vectors set to zero. After each operation, the client re-renders the workflow canvas. Users can create branches from any historical state. Each branch records its baseline version (i.e., the operation sequence number corresponding to the branch point). Operations generated independently on different branches are treated as concurrent operations, applying the same transformation rules. When a branch needs to be merged back into the main branch, the system invokes the conflict detection and semantic merging process. After all the above operations, transformations, and applications, each client will eventually receive a multi-branch workflow graph. This graph contains: all nodes that have not been deleted, all connections that have not been deleted, and merge nodes. The workflow graphs seen by all users at the same time are structurally identical.

[0039] S302 uses detection rules to identify conflict types based on the editing operation data of multiple users, merges the identified conflict types, and forms a unified workflow state; Furthermore, the detection rules include node deletion conflict detection, parameter overriding conflict detection, connection conflict detection, circular dependency conflict detection, parameter and connection intersection conflict detection, and batch operation conflict detection. Node deletion conflict detection involves deleting a node concurrently with modifying its parameters; parameter overriding conflict detection involves modifying a parameter concurrently with modifying the same parameter; connection conflict detection involves different connections from the same source port concurrently. For circular dependency conflict detection, the workflow graph is colored using depth-first search (DFS). If a directed edge from node X leads back to node X, a circular dependency is determined, and all nodes on the cycle are highlighted. The system identifies nodes and suggests adding cached nodes or deleting certain connections. For parameter and connection cross-conflict detection, when user A modifies the number of output channels of node X (e.g., from 1 to 3), and user B simultaneously connects the same output port of X to downstream node Y, which expects a single-channel input, the system detects a cross-conflict of mismatched types. For batch operation conflict detection, when user A performs a batch of copy-paste operations and user B performs deletion operations simultaneously, the system detects whether the pasted content contains nodes that have been deleted by B, based on the snapshot isolation principle. Through the above conflict type identification, the original operation data is transformed into a structured conflict type.

[0040] For detected conflicts, semantic merging is performed. The steps are as follows: For two outputs that need to be merged in parameter coverage or connection conflicts (e.g., two images with different parameters, two video clips, or two audio clips), their respective low-dimensional feature vectors are extracted—the color histogram and texture spectrum of the image, the Mel spectrum and rhythm features of the audio, and the optical flow motion features of the video. Then, the similarity score between the two feature vectors is calculated using a similarity formula. Based on the similarity level, the most suitable fusion method is selected. If the similarity is higher than 0.8, weighted stacking is recommended, where the weights are determined by the user's conflict intensity. If the similarity is between 0.4 and 0.8, content masking is recommended, that is, the image is divided into foreground / background through semantic segmentation, each taken from different sources. If the similarity is lower than 0.4, time splicing or picture-in-picture is recommended, without forced fusion. Previous merging choices made under similar conflicts are recorded, and a lightweight decision tree is constructed, that is, combining historical preferences. Personalized recommendations are implemented. For recommended merging methods, the system quickly generates low-resolution, low-bitrate side-by-side preview images in the background. Users can simply hover their mouse over conflicting nodes to see the merging effect and click to adopt the result, avoiding the consumption of full generation computing power. If a conflict cannot be merged, it is marked with a special color in the workflow diagram, and multiple sets of parameters are retained within the node. For parameters that have been automatically merged, the node parameters are directly updated to the merged values. For connection conflicts, multiple merging nodes are automatically created, and the originally conflicting connections are redirected to the merging node. For cross conflicts, conversion nodes are inserted between conflicting nodes. For circular dependencies, the system adds cache nodes or deletes connections based on user selection. For conflicts that cannot be merged, the node retains multiple parameter versions, and its metadata records the conflict flag and a list of possible values. After all the above operations, the original concurrent editing operation data of multiple users is transformed into a unified workflow state.

[0041] S303, detect the merging results of conflict types. If merging is not possible, trace the workflow status using playback and rollback strategies based on the editing operation data of multiple users; and perform interactive design on the content creation interface.

[0042] Specifically, based on the conflict type detection results, if an unmergeable conflict type is detected, the replay and rollback strategy of the collaborative operation will be invoked to trace the workflow. All editing operation data will be read from the central server. Each data entry contains a globally incrementing sequence number, parent operation sequence number, operation type, operation load, user ID, timestamp, and workflow snapshot hash. Users can use the timeline replay function to drag the slider to replay the evolution animation from the initial state to the latest state operation by operation, highlighting the nodes and connections of each modification to identify the historical causes of the conflict. If a user decides to undo a historical operation (even if the operation has been overwritten by other subsequent modifications), the inverse operation transformation algorithm will be used to calculate the minimum set of changes required to roll back from the current state to the state before the target operation. If the rollback would destroy the subsequent modifications of other users, the system will list the affected operations and provide two options: "force rollback" (discard all subsequent modifications) or "partial rollback" (rollback only the part directly affected by the operation, retaining other operations), thereby restoring the workflow state to a resolvable baseline. Meanwhile, it can create private branches from any historical operation. After users edit the branches independently, they can merge the branches back into the main branch through a merge request. During the merge, the conflict detection and semantic merge process are triggered again to supplement the asynchronous collaboration capability.

[0043] For the content creation interface, interactive design optimizations were implemented to reduce cognitive load: each online user has a unique color, the mouse cursor displays the user's avatar abbreviation, and the selected node's border displays the corresponding color; a conflict badge is added to the upper right corner of conflicting nodes, and conflict details (users involved, time, conflict type) are displayed when the mouse hovers over them; clicking the badge brings up a merge suggestion panel; the real-time collaboration sidebar lists currently online users and their recent actions (e.g., "A just deleted node X"); when the system generates a merge suggestion, a semi-transparent preview window automatically appears near the conflicting node, displaying a low-resolution thumbnail of the merge effect; users can click "Apply" to replace the merged result with a single click. These interactive designs allow team members to intuitively perceive others' actions, conflict locations, and merge effects, thereby efficiently utilizing playback and rollback functions to resolve conflicts that are difficult to merge automatically.

[0044] The technical solutions in the above embodiments of this application have at least the following technical effects or advantages: by transforming the graph through structured operations, conflict detection and semantic merging, conflict types are resolved. Conflict types that cannot be merged are resolved through replay and rollback strategies, reducing user effort consumption and enabling multiple users to drag nodes, connect lines and modify parameters simultaneously without waiting for lock release.

[0045] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A generative multimedia content creation method, characterized in that, include: S101, collect implicit behavioral data, and generate a spatiotemporal satisfaction heatmap based on the implicit behavioral data using a matrix; wherein, the implicit behavioral data includes video operation behavior data, image operation behavior data, audio operation behavior data, and text operation behavior data; S102, obtain the workflow of multimedia real-time operation, construct a data genealogy diagram based on the workflow, extract negative value voxels from the spatiotemporal satisfaction heatmap to form a problem region set, trace back along the reverse edge of the workflow according to each voxel in the problem region set to obtain the upstream node set, and identify the upstream target node that generated the problem region from the upstream node set. S103: Extract parameter types from upstream target nodes, obtain heatmap feedback features from the problem area of ​​the spatiotemporal satisfaction heatmap, and adjust adjustable parameters according to parameter types and heatmap feedback features using adjustment rules; S104, triggers re-execution based on the adjusted parameters, re-executes content creation to generate new output, compares the new output with the original output, and forms new feedback based on the comparison results; constructs a user profile database based on the adjusted parameters, and if a node of the same type is added to a new workflow, directly uses the data in the user profile database.

2. The generative multimedia content creation method according to claim 1, characterized in that, The collection of implicit behavioral data includes: setting interactive event listeners in video, image, audio, and text operation areas; setting a stillness threshold and a dwell threshold; recording a positive interest signal when the user's gaze exceeds the stillness threshold; recording a weak negative signal when the user quickly swipes over and stays for less than the dwell threshold; and classifying a segment as a positive satisfaction signal or a negative dissatisfaction signal based on whether an editing operation is performed after playback when the same segment is repeatedly played back two or more times.

3. The generative multimedia content creation method according to claim 2, characterized in that, The method for generating the spatiotemporal satisfaction heatmap is as follows: constructing a heatmap based on video operation behavior data. × × A three-dimensional matrix, in which The total number of frames in the video. For spatial resolution, construct image manipulation behavior data × Two-dimensional matrix To achieve spatial resolution, a one-dimensional matrix of length N is constructed for audio operation behavior data, and a one-dimensional matrix of length L is constructed for text operation behavior data. All matrices are initialized to 0, and their values ​​are limited to the range [-1, 1]. Negative values ​​represent implicit user dissatisfaction, while positive values ​​represent implicit satisfaction or interest.

4. The generative multimedia content creation method according to claim 3, characterized in that, The steps to generate the upstream target node of the problem area are as follows: cluster adjacent negative voxels in the heat map that are less than the preset threshold into problem areas using the connected component labeling algorithm. Starting from the currently reviewed output node, match the time index according to the modality type of the problem area, trace back along the reverse edge of the workflow, set the maximum tracing depth to 10 layers, and assign a cumulative contribution weight to each node. The node with the highest weight is taken as the upstream target node.

5. The generative multimedia content creation method according to claim 1, characterized in that, Content creation methods also include: S201, normalize the spatiotemporal satisfaction heatmap to generate a weight map, generate anchor points by setting the extreme values ​​of parameters, and weight and fuse each pixel with different anchor points according to the weight map to obtain the parameter gradient output content; S202: Divide the created content into regions to obtain several regions, configure a transition strategy for each region, perform asynchronous transitions between regions based on the transition strategy, and splice them together to obtain the spliced ​​content; wherein, the transition strategy includes transition type, transition duration, and transition curve; S203, input the output content of parameter gradient and the spliced ​​content into step S101 for review, and execute steps S101-S104.

6. The generative multimedia content creation method according to claim 5, characterized in that, The transition types include four types: fade, wipe, zoom, and dissolve; the transition duration for different regions is set independently, in seconds; the transition curve controls the rate of change during the transition process, including linear constant speed, slow in, slow out, and slow in slow out; in asynchronous transition rendering, the transition processes of different regions occur asynchronously on the timeline, and in the same frame, some regions have completed the transition, some regions are still transitioning, and some regions have not yet started transitioning.

7. The generative multimedia content creation method according to claim 5, characterized in that, The content creation method also includes: S301: Obtain editing operation data from multiple users, and construct a multi-branch workflow diagram based on the editing operation data from multiple users using operation transformation; S302 uses detection rules to identify conflict types based on the editing operation data of multiple users, merges the identified conflict types, and forms a unified workflow state; S303, detect the merging results of conflict types. If merging is not possible, trace the workflow status using playback and rollback strategies based on the editing operation data of multiple users; and perform interactive design on the content creation interface.

8. The generative multimedia content creation method according to claim 7, characterized in that, The operation transformations include: when deleting a node and modifying a parameter concurrently, retaining the deletion operation and converting the modification to an invalid operation; when different connections on the same source port are concurrently used, automatically creating a multi-path merged node; when adding a node and deleting a node concurrently, prioritizing deletion unless it is a temporary system node; when moving a node position and deleting a node concurrently, retaining the deletion and invalidating the movement; when modifying the same parameter concurrently, converting it to a combined operation carrying two candidate values ​​for semantic merging processing.

9. The generative multimedia content creation method according to claim 8, characterized in that, The semantic merging method is as follows: For two output contents that need to be merged in parameter coverage conflict or connection conflict, their respective low-dimensional feature vectors are extracted respectively: color histogram and texture spectrum of image, Mel spectrum and rhythm features of audio, and optical flow motion features of video. Then, the similarity score between the two feature vectors is calculated by similarity formula. According to the similarity, the fusion method is selected. If the similarity is higher than 0.8, weighted superposition is selected. If the similarity is between 0.4 and 0.8, content masking is selected, that is, the image is divided into foreground and background through semantic segmentation, which are taken from different sources; if the similarity is less than 0.4, time stitching or picture-in-picture is recommended, and forced fusion is not allowed.

10. A generative multimedia content creation method according to claim 9, characterized in that, The replay and rollback strategy for conflicts that cannot be merged is as follows: Edit operation data containing global sequence number, parent operation sequence number, operation type, user ID, timestamp, and workflow snapshot hash is read from the server; users can replay each operation sequentially using the timeline replay function; the minimum change set is calculated using an inverse operation transformation algorithm; during rollback, two options are provided: forced rollback discarding all subsequent modifications or partial rollback rolling back only the directly affected parts; private branches can be created from any historical operation, edited independently, and then merged back into the main branch via a merge request; during the merge, conflict detection and semantic merging are triggered again.