Atmogram library construction method and system for realizing animation materials based on big data

Abstract

The invention relates to the technical field of cartoon material library construction, and discloses a method and system for realizing cartoon material atlas library construction based on big data, and the method comprises the steps: collecting multi-element cartoon materials based on a big data technology, the multi-element cartoon materials comprise an image sequence, a structured text and unstructured audio data, and extracting image features; performing triple structure conversion on the structured text, and performing animation semantic assignment to extract text features; performing feature alignment on the audio features, the image features and the text features to construct a three-dimensional incidence matrix; and decomposing the three-dimensional incidence matrix into a multi-factor matrix, calculating the scene co-occurrence frequency of each factor in the multi-factor matrix to analyze the correlation weight of the factors in the multi-factor matrix, and constructing the map library of the multi-element animation material based on the multi-factor matrix and the correlation weight. According to the invention, the material collaboration degree of the animation map library can be improved.

Description

TECHNICAL FIELD

[0001] The application relates to a method and system for constructing an atlas library of animation materials based on big data, and belongs to the technical field of construction of an animation material library. BACKGROUND

[0002] With the rapid development of the current animation industry, with the increase of animation products, the corresponding animation materials also increase exponentially. By reasonably organizing the animation materials into an atlas library, great convenience can be brought to the calling of materials and the calling of coordinated materials according to different scenes.

[0003] Currently, the atlas library of animation materials is mainly constructed by professional personnel marking the attributes of character images, scene elements and sound effect segments, and then storing them in a database in the form of a structured table for users to query and call according to keywords. However, with the explosive growth of the animation industry, the number of materials increases exponentially, and the traditional database lacks the mining of deep semantic association between materials, cannot realize intelligent association retrieval across types and dimensions, and leads to low material collaboration degree of the animation atlas library. SUMMARY

[0004] The application provides a method and system for constructing an atlas library of animation materials based on big data, which aims to improve the material collaboration degree of the animation atlas library.

[0005] To achieve the above-mentioned purpose, the method for constructing an atlas library of animation materials based on big data provided by the application comprises the following steps: Collecting multiple animation materials based on big data technology, wherein the multiple animation materials include image sequences, structured texts and unstructured audio data, analyzing the frame entropy jump degree between each image in the image sequences, sampling the key frames of the images in the image sequences based on the frame entropy jump degree to obtain key frame images, and extracting the image features of the key frame images; Converting the structured texts into triple structure to obtain triple structure texts, assigning animation semantics to the triple structure texts to obtain domain texts, and extracting the text features of the domain texts; Performing time-frequency domain processing on the unstructured audio data to extract the audio features of the unstructured audio data, aligning the audio features, the image features and the text features to obtain aligned features, and constructing a three-dimensional correlation matrix of the multiple animation materials by using the aligned features; Decomposing the three-dimensional correlation matrix into a multi-factor matrix, calculating the scene co-occurrence frequency of each factor in the multi-factor matrix, analyzing the correlation weight of the factors in the multi-factor matrix based on the scene co-occurrence frequency, and constructing an atlas library of the multiple animation materials based on the multi-factor matrix and the correlation weight.

[0006] Optionally, the scene co-occurrence frequency of each factor in the multi-factor matrix is calculated, comprising: identifying the associated scene of each factor in the multi-factor matrix; performing factor-scene association mapping on the factors in the multi-factor matrix based on the associated scene, to obtain a factor-scene association matrix; performing time-series decay weight assignment processing on the factor-scene association matrix to obtain a weighted association matrix; calculating the scene co-occurrence frequency of each factor in the multi-factor matrix based on the weighted association matrix.

[0007] Optionally, the scene co-occurrence frequency of each factor in the multi-factor matrix is calculated based on the weighted association matrix, comprising: extracting a target factor pair in the weighted association matrix, and identifying the associated scene of the target factor pair; calculating the single-scene co-occurrence value of the target factor pair based on the associated scene; calculating the initial co-occurrence frequency of each factor in the multi-factor matrix in multiple scenes based on the single-scene co-occurrence value; performing frequency standardization processing on the initial co-occurrence frequency to obtain the scene co-occurrence frequency.

[0008] Optionally, a three-dimensional association matrix of the multi-element animation material is constructed using the alignment features, comprising: constructing a three-order tensor slice of the multi-element animation material using the alignment features; performing high-order singular value decomposition on the three-order tensor slice, and performing singular value compression processing on the decomposed three-order tensor slice to obtain a low-rank core tensor; performing association strength quantization processing on the low-rank core tensor to obtain an association core tensor; constructing a three-dimensional association matrix of the multi-element animation material using the association core tensor.

[0009] Optionally, the frame entropy jump degree between each image in the image sequence is analyzed, comprising: convert each image in the image sequence into a grayscale image to obtain a set of grayscale images; statistically analyzing the grayscale histogram of each image in the set of grayscale images; calculating the single-frame entropy of the image corresponding to the grayscale histogram; calculating the frame entropy jump degree of adjacent images in the set of grayscale images based on the single-frame entropy.

[0010] Optionally, animation semantic assignment is performed on the triple structure text to obtain a domain text, comprising: Perform role semantic annotation on the text entity corresponding to the triple structure text to obtain an annotated text; Perform ontology mapping based on the animation field on the annotated text to obtain a subject mapping text; Perform multi-level semantic enhancement processing on the subject mapping text to obtain an enhanced structure text; Perform field-based text reconstruction on the enhanced structure text to obtain a field-based text.

[0011] Optionally, the unstructured audio data is subjected to time-frequency domain processing to extract audio features of the unstructured audio data, including: The unstructured audio data is subjected to pre-emphasis processing to obtain enhanced audio; The enhanced audio is subjected to frame windowing processing to obtain segmented audio; The segmented audio is subjected to short-time Fourier transform to obtain a time-frequency domain spectrogram; Mel-frequency cepstral coefficients of the time-frequency domain spectrogram are extracted, and the energy, zero-crossing rate and fundamental frequency of each frame of audio in the time-frequency domain spectrogram are calculated; The Mel-frequency cepstral coefficients are combined with the energy, the zero-crossing rate and the fundamental frequency to obtain audio features of the unstructured audio data.

[0012] Optionally, the audio features, the image features and the text features are subjected to feature alignment to obtain aligned features, including: The audio features, the image features and the text features are subjected to cross-modal vector space projection processing to obtain a feature vector group; The feature vector group is subjected to reinforcement learning to obtain an optimized feature vector; The optimized feature vector is subjected to triple dynamic constraint to obtain a boundary-separated feature vector; The boundary-separated feature vector is subjected to graph attention relationship fusion processing to obtain aligned features.

[0013] Optionally, based on the multi-factor matrix and the correlation weight, a graph library of the multi-element animation material is constructed, including: The multi-factor matrix is used to construct a graph node of the multi-element animation material; Based on the correlation weight, the graph node is subjected to node linking to obtain a graph grid; The graph grid is used to construct a graph library of the multi-element animation material.

[0014] To solve the above problems, the application further provides a system for constructing a graph library of animation materials based on big data, which comprises: An image material processing module is configured to collect multi-element animation materials based on big data technology, the multi-element animation materials including image sequences, structured texts and unstructured audio data, analyze frame entropy jump degrees between each image in the image sequences, perform visual key frame sampling on images in the image sequences based on the frame entropy jump degrees to obtain key frame images, and extract image features of the key frame images; A text material processing module is configured to perform triple structure conversion on the structured texts to obtain triple structure texts, perform animation semantic assignment on the triple structure texts to obtain field texts, and extract text features of the field texts; An association matrix construction module is configured to perform time-frequency domain processing on the unstructured audio data to extract audio features of the unstructured audio data, perform feature alignment on the audio features, the image features and the text features to obtain aligned features, and construct a three-dimensional association matrix of the multi-element animation materials by using the aligned features. A graph library construction module is configured to decompose the three-dimensional association matrix into multi-factor matrices, calculate scene co-occurrence frequencies of each factor in the multi-factor matrices, analyze association weights of the factors in the multi-factor matrices based on the scene co-occurrence frequencies, and construct a graph library of the multi-element animation materials based on the multi-factor matrices and the association weights.

[0015] The application first integrates the animation materials (image sequence, text, audio) of the production end, the transmission end and the user end through big data technology, ensures the comprehensiveness and diversity of data coverage, then accurately identifies the key frames of visual information mutation in the image sequence, such as role transformation, skill release and other scenes, by means of frame entropy jump degree analysis, not only eliminates repeated frames to reduce data redundancy, but also completely retains the core visual content. Further, the application converts the structured text into a "subject-relation-object" triple to clearly comb the logical relationship between entities, solves the problem of fuzzy associated information in the original text, and further assigns semantics in the animation field (makes the text fit the animation scene context, realizes the quantitative expression of text information, provides semantic anchor points for the association of image and audio features, and enhances the matchability of cross-modal data. Further, the application converts the audio signal into quantifiable data, such as distinguishing character voice from background sound effects through fundamental frequency, and aligning audio, image and text features to a unified vector space through cross-modal vector projection and reinforcement learning, and presenting the correlation strength of multi-element materials in a structured form, providing a structured data basis for subsequent factor decomposition. Further, the application decomposes the three-dimensional correlation matrix into a multi-factor matrix, extracts independent feature factors of audio, image and text, simplifies the data structure while highlighting the single-modal core features, and calculates the scene co-occurrence frequency of the factors to quantify the factor correlation rules in the scene and determine the correlation strength between factors. Therefore, the application can improve the material collaboration degree of the animation atlas library. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The flowchart of the method for constructing an atlas library of animation materials based on big data provided by an embodiment of the application is shown. Figure 2 The module diagram of the method for constructing an atlas library of animation materials based on big data provided by an embodiment of the application is shown.

[0017] The purpose of the application, functional features and advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0018] It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application.

[0019] This application provides a method for constructing an anime material atlas library based on big data. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method for constructing an anime material atlas library based on big data can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0020] Reference Figure 1 The diagram shown is a flowchart illustrating a method for constructing an anime material library based on big data, according to an embodiment of the present invention. In this embodiment, the method for constructing an anime material library based on big data includes: S1. Collect diverse animation materials based on big data technology. The diverse animation materials include image sequences, structured text, and unstructured audio data. Analyze the frame entropy jump degree between each image in the image sequence. Based on the frame entropy jump degree, perform visual keyframe sampling on the images in the image sequence to obtain keyframe images, and extract the image features of the keyframe images.

[0021] This invention integrates diverse animation materials from multiple channels, including production, distribution, and user terminals, by collecting them using big data technology. This ensures comprehensive and diverse data coverage, thereby providing reliable data support for the construction of an animation material library.

[0022] The diverse animation materials include image sequences, structured text, and unstructured audio data, which can be obtained through web crawlers, API interfaces under user authorization agreements, or by scanning physical animations using digital technology.

[0023] Furthermore, by analyzing the frame entropy jump between each image in the image sequence, this embodiment of the invention can understand the degree of abrupt changes in pixel distribution, color composition, and content complexity between adjacent images.

[0024] The frame entropy jump degree is an indicator used in video processing to measure the degree of information change between consecutive frames, and is mainly used to identify scene switching or key action nodes.

[0025] As an embodiment of the present invention, analyzing the frame entropy jump degree between each image in the image sequence includes: Each image in the image sequence is converted into a grayscale image to obtain a grayscale image set; Calculate the grayscale histogram for each image in the grayscale image set; Calculate the single-frame entropy of the image corresponding to the grayscale histogram; Based on the single-frame entropy, the frame entropy jump degree of adjacent images in the grayscale image set is calculated.

[0026] In practice, a weighted average method can be used to convert color images in anime image sequences into grayscale images. For each image in the grayscale image set, the OpenCV calcHist function can be used to count the percentage of pixels at each grayscale level from 0 to 255. The absolute difference ΔH = |H(n+1) - H(n)| is taken for the single-frame entropy of adjacent grayscale images. For example, in the adjacent frames from when a character is still to when he activates a skill, H(n) = 3.1, H(n+1) = 7.9, ΔH = 4.8. The threshold is set to the mean of ΔH in the sequence (e.g., 2.5) plus 1.2 times the standard deviation (e.g., 1.8), that is, threshold = 4.3. If it exceeds this threshold, it is marked as a significant jump.

[0027] Furthermore, as another embodiment of the present invention, the formula for calculating single-frame entropy is as follows: ; in, Represents the entropy of a single frame. This represents the percentage of pixels at the i-th gray level of the image corresponding to the gray-level histogram.

[0028] Furthermore, this embodiment of the invention performs visual keyframe sampling on the images in the image sequence based on the frame entropy jump degree. This results in keyframe images that can accurately capture frames where visual information changes significantly from a continuous stream of anime images. This avoids data redundancy caused by repeated or similar frames while fully preserving the core visual content of the sequence. For example, in anime character transformation scenes, this step can filter out frames with obvious differences in form before and after the transformation, without retaining the subtle transition frames during the transformation process, allowing subsequent processing to focus on truly valuable visual information.

[0029] The keyframe images refer to image frames selected from animation image sequences that exhibit significant changes in visual information and carry core content. These frames reflect important visual information such as scene transitions, character morphological changes, and key action nodes in the image sequence, and are the most representative image units in the image sequence.

[0030] In practice, a frame entropy jump threshold can be set first (e.g., 12 after statistical analysis). The ΔH of adjacent frames is calculated by traversing the image sequence: when ΔH > 12, the next frame is marked as a keyframe; when ΔH ≤ 12 for multiple consecutive frames, one frame is retained as a keyframe every five frames. For example, in 100 frames of a running scene in an anime, the character's starting action in the first 10 frames causes ΔH to exceed 15, so frames 2-10 are marked as keyframes. In the middle 60 frames, the character runs at a constant speed, and ΔH stabilizes at around 8. One frame is retained every five frames, resulting in 12 frames. In the last 30 frames, the character stops, and ΔH jumps from 9 to 14. The last five frames are marked as keyframes. Finally, 27 keyframe images are sampled from the 100 frames.

[0031] Furthermore, by extracting the image features of the keyframe image, the present invention can extract information that represents its visual characteristics, such as character outlines and scene colors, so that the core visual content of the keyframe can be quantitatively analyzed. For example, features such as weapon shape and attack posture can be extracted from the keyframe of anime character battle to accurately present the visual core of the frame.

[0032] Optionally, the image features can be extracted using a convolutional model in machine learning.

[0033] S2. Perform triple structure transformation on the structured text to obtain triple structured text, assign animation semantic values ​​to the triple structured text to obtain domain-specific text, and extract the text features of the domain-specific text.

[0034] This invention transforms the structured text into a triplet structure by performing a triplet transformation, which converts the structured text into a "subject-relationship-object" triplet form, making the relationship between entities in the text clearer and more intuitive. For example, "Luffy uses the rubber fruit" is transformed into (Luffy, uses, rubber fruit), accurately presenting the core relationship between the character and the item.

[0035] The triplet structured text refers to structured data presented in the form of "subject-relationship-object", where "subject" and "object" are entities with clear orientations, and "relationship" is the action, attribute or association that connects the two.

[0036] In practice, entities (such as characters and props) and relationships between entities (such as actions and ownership) can be identified from structured text first, and then reorganized in the format of "subject-relationship-object". For example, "Pikachu releases 100,000 volts" can be converted into (Pikachu, release, 100,000 volts) by extracting the subject "Pikachu", the relationship "releases", and the object "100,000 volts".

[0037] Furthermore, by assigning anime semantic values ​​to the triplet structure text, the resulting domain-specific text can endow entities and relations in the triplet with anime-specific semantics, making the text more in line with anime scenes. For example, assigning (Luffy, use, ability) to (Luffy, activate, rubber fruit ability) clarifies the specific meaning of "ability" in anime.

[0038] The term "domain-specific text" refers to text that is adjusted or supplemented to better fit the context of a specific domain (such as animation, medicine, or finance) by incorporating professional knowledge, terminology, and scene characteristics of the basic text.

[0039] As an embodiment of the present invention, the triplet structure text is assigned animation semantic values ​​to obtain domain-specific text, including: Role semantic annotation is performed on the text entities corresponding to the triplet structure text to obtain the annotated text; Perform ontology mapping on the annotated text based on the animation domain to obtain the main mapped text; The main mapped text is subjected to multi-level semantic enhancement processing to obtain enhanced structured text; The enhanced structural text is reconstructed using domain-specific text to obtain domain-specific text.

[0040] The labeled text refers to the text after adding anime-specific semantic tags to entities (such as characters, skills, props, etc.) in the triplet structure text, and the subject mapping text refers to the text after associating and matching entities in the labeled text with standardized concepts in the anime domain ontology library.

[0041] In practice, a BERT pre-trained model can be used to perform named entity recognition on text entities in triples, labeling them with their role type in the anime (such as protagonist, villain, support character, etc.). For example, "Sasuke" in the triple (Sasuke, uses, Sharingan) can be labeled as "Uchiha clan villain (later becomes a hero)", forming labeled text. Based on a pre-set anime domain ontology library (containing 20 core categories such as characters, skills, and items), the Protege tool is used to map entities in the labeled text to the corresponding ontology categories. For example, "Sharingan" can be mapped to the "Bloodline Limit Skill" category, resulting in the ontology mapping text. The TransE knowledge graph completion model is used to supplement semantic information from three levels: skill attributes (such as cooldown time, power level), scene restrictions (such as battle / daily scenes), and associated characters (the specific levels need to be set according to the actual application scenario or user needs). The anime script expression specifications are followed (such as "[character type] [character name] [skill attributes]"). "Activate [Skill Category]", which will enhance the restructuring of the structured text, for example, reconstruct it as "Uchiha clan villain (later turned to the good guys) Sasuke activates the Sharingan, a bloodline limit skill with a power level of A+ (only in battle scenes, associated character: Itachi)", forming a domain-specific text.

[0042] Furthermore, embodiments of the present invention can extract features from the domain-specific text that reflect core information of the animation domain, such as character identifiers and skill attributes, thereby enabling the text information to be quantitatively analyzed.

[0043] Optionally, the text features can be obtained by inputting domain-specific text into a natural language model and extracting them using the natural language model.

[0044] S3. Perform time-frequency domain processing on the unstructured audio data to extract the audio features of the unstructured audio data, align the audio features, the image features, and the text features to obtain aligned features, and use the aligned features to construct a three-dimensional association matrix of the multi-dimensional animation materials.

[0045] This invention provides an embodiment of the invention that extracts audio features from unstructured audio data by performing time-frequency domain processing on the unstructured audio data. Key information such as pitch, rhythm, and spectral features can be extracted from unstructured audio, making it quantifiable and analytic. For example, frequency features of weapon collisions and pitch changes of character shouts can be extracted from the audio of anime battle scenes.

[0046] As an embodiment of the present invention, the unstructured audio data is subjected to time-frequency domain processing to extract audio features of the unstructured audio data, including: The unstructured audio data is pre-emphasized to obtain enhanced audio; The enhanced audio is then subjected to frame-by-frame windowing processing to obtain segmented audio; Perform a short-time Fourier transform on the segmented audio to obtain a time-frequency domain spectrum. Extract the Mel frequency cepstral coefficients from the time-frequency domain spectrogram, and calculate the energy, zero-crossing rate, and fundamental frequency of each frame of audio in the time-frequency domain spectrogram; The audio features of unstructured audio data are obtained by combining the Mel frequency cepstral coefficients with the energy, the zero-crossing rate, and the fundamental frequency.

[0047] The enhanced audio refers to the audio signal obtained after pre-emphasizing unstructured audio data (such as fighting sound effects and character lines in anime). The segmented audio refers to the set of short-duration audio segments formed after the enhanced audio is processed by frame segmentation and windowing. The time-frequency domain spectrogram refers to the two-dimensional visualization image obtained after performing a short-time Fourier transform on the segmented audio. The horizontal axis represents time, the vertical axis represents frequency, and the gray level or color depth of the pixels reflects the signal energy at the corresponding time-frequency point.

[0048] In practice, a first-order high-pass filter (with a transfer function of H(z) = 1 - 0.97) can be used. The unstructured audio data is processed to enhance high-frequency components above 2kHz, such as enhancing the high-frequency details of a character's "shout" in anime, while suppressing low-frequency background noise, resulting in enhanced audio. The enhanced audio is then divided into 25ms / frame segments (with a 50% overlap, meaning each frame slides for 12.5ms). A Hanning window (window function w(n) = 0.5 - 0.5cos²) is applied to each frame. (2πn / (N-1)), where N is the frame length of 512 points), for example, the 1-second sound effect of "skill release" in anime is divided into 40 frames to reduce spectral leakage and obtain segmented audio; 13th-order Mel frequency cepstral coefficients (MFCCs) are extracted from the spectrogram, which can be obtained by mapping 26 Mel filter banks and then performing discrete cosine transform (DCT). At the same time, the energy (sum of squares of the signal within the frame), zero-crossing rate (the number of times the signal crosses zero per second, with a threshold of ±0.01 amplitude) and fundamental frequency of each frame are calculated (autocorrelation method can be used, with a range of 50-500Hz, suitable for the vocal range of anime character voice acting); the 13th-order MFCCs are concatenated with the energy, zero-crossing rate, and fundamental frequency frame by frame to form a feature vector of dimension 16. For example, the audio of "character dialogue + background music" in anime, after combination, contains both the MFCC features of speech and the energy fluctuations and zero-crossing rate features of background music, forming a complete audio feature.

[0049] Furthermore, in this embodiment of the invention, by aligning the audio features, the image features, and the text features, the aligned features can accurately match the features of audio, image, and text in the temporal or semantic dimensions, allowing different modal information to form a connection. For example, in anime, the text of a character "releasing a skill," the keyframe image of the skill's light effect, and the audio features of the skill's sound effect can be aligned.

[0050] As an embodiment of the present invention, feature alignment is performed on the audio features, the image features, and the text features to obtain aligned features, including: The audio features, image features, and text features are subjected to cross-modal vector space projection processing to obtain a feature vector group; Reinforcement learning is performed on the feature vector group to obtain optimized feature vectors; The optimized feature vector is subjected to triplet dynamic constraints to obtain the boundary separation feature vector; The boundary separation feature vectors are subjected to graph attention relationship fusion processing to obtain aligned features.

[0051] The feature vector group refers to the set of vectors formed by projecting audio features, image features, and text features onto the same high-dimensional vector space through a cross-modal mapping model (such as the CLIP model based on contrastive learning). The boundary separation feature vector refers to the vector obtained by applying a triplet loss function (such as Triplet Loss with margin=0.5) to the feature vector group for dynamic constraint.

[0052] In practice, pre-trained models can be used to convert the features of each modality into initial vectors (768-dimensional vectors for text, 2048-dimensional vectors for images, and 128-dimensional vectors for audio using VGGish). These vectors are then mapped to a unified 512-dimensional vector space using the learned projection matrix. For example, the text description of "Luffy's Gum-Gum Fist," close-up images of the fist, and the stretching sound effects from the anime can be projected into the same space, making cross-modal features comparable. A reinforcement learning framework (using the DQN algorithm) is constructed with "intra-modal consistency" and "inter-modal correlation" as reward functions. A +10 reward is given when the cosine similarity of different modal vectors is ≥0.7, and a -10 reward is given when the cosine similarity is ≤0.3. 5. Penalty: Vector optimization is achieved through iterative training. For example, in the scenario of "Sasuke's Sharingan activation," the vector is continuously adjusted to increase the similarity between the eye technique image features and the text features of "Mangekyou Sharingan" from 0.5 to 0.85. An online triplet mining strategy is adopted, randomly selecting anchor features (such as the sound effect vector of a certain skill), positive samples (the light effect image vector of the same skill), and negative samples (the text vector of other skills) in each round. By dynamically adjusting the margin value (set to 0.5±0.1, adaptively changing according to the sample distribution), the L2 distance between the anchor and the positive sample is ≤0.3. Furthermore, the distance from negative samples is ≥0.8, which strengthens the boundary separation of different semantic features. For example, it ensures that the feature vectors of "Kamehameha" and "Spirit Bomb" are significantly separated. A graph structure with each modality feature as a node is constructed, and GAT (Graph Attention Network) is used to calculate the edge weights. The attention coefficient (normalized by softmax, with a value range of 0-1) is used to highlight strongly correlated features. For example, the attention coefficient between the audio feature of "character roar" and the facial expression image feature is set to 0.8, and the coefficient with the irrelevant scene text is set to 0.1. Finally, all node features are fused to obtain the aligned features.

[0053] Furthermore, by utilizing the alignment features, this embodiment of the invention constructs a three-dimensional association matrix of the diverse animation materials, which can structurally present the association relationships of the aligned audio, image, and text features, clearly quantifying the intrinsic connections between the diverse animation materials. For example, in an animation "character transformation" scene, the alignment features of the transformation image, the "Transform!" text, and the transformation sound effect are reflected in the matrix through high association values, demonstrating the strong binding relationship between the three.

[0054] The three-dimensional correlation matrix refers to a structured data model that uses audio features, image features, and text features as three dimensions and quantifies the correlation strength between different modal features through matrix unit values.

[0055] As an embodiment of the present invention, the three-dimensional association matrix of the multi-dimensional animation materials is constructed using the alignment feature, including: Using the alignment features, a third-order tensor slice of the diverse animation material is constructed; The third-order tensor slices are subjected to high-order singular value decomposition, and the decomposed third-order tensor slices are subjected to singular value compression to obtain low-rank core tensors. The low-rank core tensor is subjected to correlation strength quantization to obtain the correlation core tensor; Using the aforementioned core relational tensor, a three-dimensional relational matrix of the diverse animation materials is constructed.

[0056] The third-order tensor slice refers to a sub-tensor block formed by dividing the alignment features into three dimensions: audio, image, and text features, according to timestamps or semantic logic (such as animation scene clips). The low-rank core tensor refers to a compressed tensor obtained by performing high-order singular value decomposition on the third-order tensor slice and retaining the main singular values ​​with a cumulative energy ratio of 90% (such as removing redundant values ​​with an energy ratio of <1%). The associated core tensor refers to a tensor that reflects the correlation strength between audio, image, and text features.

[0057] In practice, the audio, image, and text features in the alignment features can be used as the three dimensions of a third-order tensor (e.g., audio frames, image keyframes, and text sentence vectors). Sub-features under each dimension are grouped by timestamp or semantics to form several tensor slices. For example, in the anime "magic casting" scene, the incantation audio clips, magic light effect image frames, and the text sentence "fireball cast" segmented at 100ms intervals together constitute a third-order tensor slice of the scene (this can be implemented using TensorFlow tools). The third-order tensor slice is then decomposed using the High-Order Singular Value Decomposition (HOSVD) algorithm to obtain the factor matrix and core tensor. Singular values ​​with a cumulative energy percentage of 90% are retained (e.g., the first 30 of the original 100-order singular values ​​are retained). Then, compression is achieved by truncating the singular values ​​to obtain a low-rank core tensor. For example, after decomposing the tensor slice of the "battle scene,"... Redundant singular values ​​with an energy percentage of less than 1% are removed to reduce the amount of data while retaining core correlation information. The element values ​​of the low-rank core tensor are normalized (using the min-max method). The closer the value is to 1, the stronger the correlation of the corresponding dimension features. A correlation strength threshold of 0.7 can be set. Elements exceeding the threshold are marked as strongly correlated. For example, the element value of the tensor "character lines - facial expression images - tone audio" is 0.85 after quantization, which is determined to be strongly correlated. The correlation core tensor is converted into a three-dimensional matrix form. The matrix coordinates (i, j, k) correspond to the audio feature index, image feature index, and text feature index, respectively. The coordinate values ​​are the quantized correlation strength. For example, in the three-dimensional matrix of the anime "transformation scene", the coordinate value of (5, 12, 8) 0.92 indicates that there is a strong correlation between the 5th transformation sound effect, the 12th frame transformation image, and the 8th transformation line.

[0058] S4. Decompose the three-dimensional correlation matrix into a multi-factor matrix, calculate the scene co-occurrence frequency of each factor in the multi-factor matrix, analyze the correlation weight of the factors in the multi-factor matrix based on the scene co-occurrence frequency, and construct the multi-dimensional animation material atlas library based on the multi-factor matrix and the correlation weight.

[0059] This invention decomposes the three-dimensional correlation matrix into a multi-factor matrix to extract independent feature factors of each modality (audio, image, text) and correlation factors between modalities. This simplifies the data structure while highlighting the core features of each dimension. For example, the three-dimensional correlation matrix of "battle scene" is decomposed into weapon collision sound effect factors, sword swinging action image factors, etc., clearly presenting the single-modal feature patterns.

[0060] The multi-factor matrix refers to a set of factor matrices formed by the three types of single-modal features of audio, image and text in the animation material, where each factor matrix corresponds to the feature pattern of a modality.

[0061] In practice, the three dimensions of the three-dimensional correlation matrix—audio, image, and text—can be used as a benchmark. The core factors to be extracted for each dimension (such as "sound effect type" for audio, "scene action" for image, and "dialogue intent" for text) can be clearly defined. Then, the non-negative matrix factorization (NMF) algorithm is used to decompose the projection of the matrix onto each dimension, resulting in a multi-factor matrix. For example, for the three-dimensional matrix of "battle scene," the audio dimension can be decomposed into factors such as "weapon collision" and "character shouts," while the image dimension can be decomposed into factors such as "sword swinging action" and "explosion effects."

[0062] This invention quantifies the scene association patterns of feature factors by calculating the scene co-occurrence frequency of each factor in the multi-factor matrix. For example, in the "magic battle" scene, calculating the co-occurrence frequency of "spell audio factor" and "magic light effect image factor" can clearly identify them as a typical feature combination of the scene.

[0063] The scene co-occurrence frequency refers to the statistical probability used to quantify the simultaneous occurrence of different elements (such as words, visual objects, audio features, etc.) in a specific scene in a spatial, temporal, or semantic context.

[0064] As an embodiment of the present invention, calculating the scene co-occurrence frequency of each factor in the multi-factor matrix includes: Identify the associated scenarios for each factor in the multi-factor matrix; Based on the aforementioned associated scenarios, factor-scenario association mapping is performed on the factors in the multi-factor matrix to obtain the factor-scenario association matrix; The factor-scene correlation matrix is ​​subjected to time-series decay weight assignment to obtain a weighted correlation matrix; Based on the weighted correlation matrix, the scene co-occurrence frequency of each factor in the multi-factor matrix is ​​calculated.

[0065] The associated scene refers to the animation plot scene to which each feature factor (such as audio, image, and text factors) in the multi-factor matrix belongs. The factor-scene association matrix is ​​a two-dimensional matrix that records the correspondence between factors and associated scenes. The weighted association matrix is ​​the matrix obtained by assigning time-decay weights to the factor-scene association matrix.

[0066] In practice, scene label matching algorithms (such as BERT-based text similarity matching) can be used to associate each factor in the multi-factor matrix with a preset scene library (such as "battle", "dialogue", "magic duel"). A similarity threshold (such as 0.8) is set. When the matching degree between the factor feature and the scene label is greater than or equal to the threshold, it is determined to be an associated scene of that factor. A factor-scene association matrix with the dimension of "number of factors × number of scenes" is constructed. The matrix element values ​​represent the association confidence of the factor and the scene (range 0-1). The mapping can be achieved by combining One-Hot encoding with confidence weighting. For example, the association confidence of "magic light effect image factor" in "magic duel scene" is 0.9, and the corresponding position in the matrix is ​​recorded as 0.9. The confidence of "everyday dialogue scene" is 0.1, and it is recorded as 0.1. For each element in the factor-scene association matrix, the weighted association matrix is ​​obtained by attenuating the weight according to the duration of the factor's appearance time in the scene from the scene's starting point. The following formula can be used for assignment: ( The attenuation coefficient is set to 0.05 / second. (Initial association confidence level).

[0067] Furthermore, as an optional embodiment of the present invention, calculating the scene co-occurrence frequency of each factor in the multi-factor matrix based on the weighted correlation matrix includes: Extract the target factor pairs from the weighted correlation matrix and identify the associated scenarios of the target factor pairs; Based on the associated scenarios, calculate the single-scenario co-occurrence value of the target factor pair; Based on the single-scene co-occurrence value, calculate the initial co-occurrence frequency of each factor in the multi-factor matrix in multiple scenarios; The initial co-occurrence frequency is normalized to obtain the scene co-occurrence frequency.

[0068] Wherein, the target factor pair refers to the combination of two feature factors selected from the multi-factor matrix to analyze the correlation strength; the single-scene co-occurrence value refers to the numerical value that quantifies the co-occurrence strength of the target factor pair by calculating the number of co-occurrences and factor weights in a single associated scenario; and the initial co-occurrence frequency refers to the original frequency value obtained by weighted summation of the single-scene co-occurrence values ​​of the target factor pair in multiple associated scenarios.

[0069] In practice, factor pairs to be calculated (such as "magic chanting audio factor i" and "glowing wand image factor j") can be selected from the weighted association matrix. Through the weight mapping relationship between factors and scenes in the matrix, scenes that are jointly associated and have weights ≥ 0.3 are selected as valid associated scenes. For each associated scene, the number of times the factor pair co-occurs in the scene is statistically analyzed by comparing timestamps. Combining the weights of factor i and factor j in the scene, the co-occurrence value of a single scene is calculated according to the formula Vᵢⱼ = Nᵢⱼ × Wᵢ × Wⱼ. The initial co-occurrence frequency is then z-score standardized to obtain the scene co-occurrence frequency of each factor in the multi-factor matrix.

[0070] Furthermore, as another embodiment of the present invention, the formula for calculating the initial co-occurrence frequency is as follows: ; in, Indicates the initial co-occurrence frequency. This indicates the number of scenarios in the weighted association matrix. This represents the single-scene co-occurrence value of factor pair (i, j) in the s-th scene. Indicates the first The importance weight of each scenario This represents the weight value of factor i in the s-th scene. This represents the weight value of factor j in the s-th scenario.

[0071] It should be noted that the above formula for calculating the initial co-occurrence frequency comprehensively considers the co-occurrence strength of factors in various related scenarios, takes into account the differences in importance of different scenarios, weights and sums the co-occurrence values ​​of individual scenarios according to scenario importance, and then normalizes them with the basic association strength of factors in the scenario. This integrates information from multiple scenarios to form an initial frequency that reflects the overall co-occurrence trend of factors. This formula highlights the dominant role of factor co-occurrence in core scenarios, weakens the interference of secondary scenarios, and balances the weight differences of different scenarios through normalization, making the calculation results more consistent with the association nature of factors in actual application scenarios, thus improving the accuracy and discriminative power of the co-occurrence frequency.

[0072] Furthermore, this embodiment of the invention can quantify the importance of the correlation between factors in the multi-factor matrix by analyzing the correlation weight of factors in the multi-factor matrix based on the co-occurrence frequency of the scene, so that the correlation strength has a clear weight value. For example, in the anime battle scene, the co-occurrence frequency of "weapon collision audio factor" and "character sword swinging image factor" reaches 80%, and after analysis, its correlation weight is significantly higher than that of other factors, highlighting the core correlation between the two in the battle scene.

[0073] The association weight refers to a numerical parameter that quantifies the degree of association between different modal feature factors in the same animation scene based on the scene co-occurrence frequency of factors in the multi-factor matrix.

[0074] In practice, the co-occurrence frequency of a scene can be mapped to the [0, 1] interval using a normalization algorithm. The resulting value is the correlation weight between the corresponding factors in the multi-factor matrix. For example, the co-occurrence frequency of 0.7 between the "magic chanting audio factor" and the "light effect image factor" is normalized and used as the correlation weight between the two.

[0075] Furthermore, by constructing a graph library of diverse animation materials based on the multi-factor matrix and the associated weights, the embodiments of the present invention can enable animation materials to form an associated network in the graph library that fits the scene logic. For example, in a "battle scene", the weapon collision sound effect factor and the sword swing image factor form a corresponding relationship according to the weights, so that different types of materials can naturally echo each other in the scene presentation.

[0076] The graph library refers to a structured collection that presents various animation-related materials (such as audio clips, image frames, text lines, etc.) or their core features in the form of nodes. The connection relationship between nodes reflects the relationship between different materials in terms of content, scene or logic, and can intuitively show the inherent connection between materials.

[0077] As an embodiment of the present invention, a graph library of multi-dimensional animation materials is constructed based on the multi-factor matrix and the correlation weights, including: Using the multi-factor matrix, construct the graph nodes of the multi-dimensional animation materials; Based on the association weights, the graph nodes are linked together to obtain a graph mesh; Using the aforementioned graph grid, a graph library of the diverse animation materials is constructed.

[0078] The graph node refers to the concrete unit of the audio, image, and text single-modal feature factors extracted from the multi-factor matrix. Each node corresponds to one feature factor. The graph grid refers to the network structure formed by connecting the graph nodes through association weights.

[0079] In practice, audio, image, and text feature factors can be extracted from a multi-factor matrix. Each factor is assigned a unique identifier (e.g., audio factor ID "AF001", image factor ID "IF015") as the basic node of the graph library. A correlation weight threshold (e.g., 0.6) is set. When the correlation weight between two nodes of different modalities exceeds the threshold, a weighted edge is created using the graph database's relationship creation function. The edge's attribute value is the correlation weight. For example, the correlation weight between "spell audio factor" and "magic light effect image factor" is 0.85. A connection edge with a weight of 0.85 is created between them, forming a preliminary graph grid. The graph grid is imported into the graph database. Index optimization (e.g., indexing node IDs and weight values) improves the efficiency of correlation queries. Isolated nodes in the grid (nodes with correlation weights all below 0.3) are marked or have supplementary correlations added, ultimately forming a multi-dimensional animation material graph library containing complete node relationships. For example, in the graph library for the "magic battle" scene, all highly correlated nodes are connected by weighted edges, allowing for quick querying of all image and audio nodes associated with the "spell text factor".

[0080] like Figure 2 The diagram shown is a functional module diagram of the system for constructing an animation material image library based on big data, according to the present invention.

[0081] The big data-based anime material graph library construction system 200 described in this invention can be installed in an electronic device. Depending on the functions implemented, the big data-based anime material graph library construction system may include an image material processing module 201, a text material processing module 202, an association matrix construction module 203, and a graph library construction module 204. The modules described in this invention can also be called units, referring to a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0082] In this embodiment of the invention, the functions of each module / unit are as follows: The image material processing module 201 is used to collect diverse animation materials based on big data technology. The diverse animation materials include image sequences, structured text, and unstructured audio data. It analyzes the frame entropy jump degree between each image in the image sequence, performs visual keyframe sampling on the images in the image sequence based on the frame entropy jump degree, obtains keyframe images, and extracts the image features of the keyframe images. The text material processing module 202 is used to perform triple structure transformation on the structured text to obtain triple structure text, assign animation semantic values ​​to the triple structure text to obtain domain-specific text, and extract the text features of the domain-specific text. The association matrix construction module 203 is used to perform time-frequency domain processing on the unstructured audio data to extract the audio features of the unstructured audio data, align the audio features, the image features, and the text features to obtain aligned features, and use the aligned features to construct a three-dimensional association matrix of the multi-dimensional animation materials. The graph library construction module 204 is used to decompose the three-dimensional correlation matrix into a multi-factor matrix, calculate the scene co-occurrence frequency of each factor in the multi-factor matrix, analyze the correlation weight of the factors in the multi-factor matrix based on the scene co-occurrence frequency, and construct the graph library of the multi-dimensional animation materials based on the multi-factor matrix and the correlation weight.

[0083] In detail, the modules in the big data-based animation material library construction system 200 described in this embodiment of the invention employ the same methods as described above. Figure 1 The method described above uses the same techniques as the big data-based method for constructing anime material image libraries, and can produce the same technical effects, so it will not be elaborated here.

[0084] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0085] Finally, it should be noted that in the above embodiments, each embodiment can be combined with each other or independent. Deleting any one of them will not affect the technical implementation of other embodiments. The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for constructing an atlas library of animation materials based on big data, characterized in that, The method comprises: Based on big data technology, multi-element animation materials are collected, the multi-element animation materials include image sequences, structured text and unstructured audio data, frame entropy jump degrees between each image in the image sequences are analyzed, based on the frame entropy jump degrees, key frames in the image sequences are sampled to obtain key frame images, and image features of the key frame images are extracted; The structured text is converted into a triple structure to obtain triple structure text, animation semantic assignment is performed on the triple structure text to obtain domain text, and text features of the domain text are extracted; The unstructured audio data is processed in time and frequency domains to extract audio features of the unstructured audio data, the audio features, the image features and the text features are aligned to obtain aligned features, and a three-dimensional correlation matrix of the multi-element animation materials is constructed by using the aligned features; The three-dimensional correlation matrix is decomposed into a multi-factor matrix, scene co-occurrence frequencies of each factor in the multi-factor matrix are calculated, correlation weights of the factors in the multi-factor matrix are analyzed based on the scene co-occurrence frequencies, and a graph library of the multi-element animation materials is constructed based on the multi-factor matrix and the correlation weights. 2.The method of claim 1, wherein, The calculation of the scene co-occurrence frequencies of each factor in the multi-factor matrix comprises: Identifying correlation scenes of each factor in the multi-factor matrix; Based on the correlation scenes, factor-scene correlation mapping is performed on the factors in the multi-factor matrix to obtain a factor-scene correlation matrix; The factor-scene correlation matrix is subjected to time sequence attenuation weight assignment processing to obtain a weighted correlation matrix; Based on the weighted correlation matrix, the scene co-occurrence frequencies of each factor in the multi-factor matrix are calculated. 3.The method of claim 2, wherein, The calculation of the scene co-occurrence frequencies of each factor in the multi-factor matrix based on the weighted correlation matrix comprises: Extracting a target factor pair in the weighted correlation matrix and identifying correlation scenes of the target factor pair; Based on the correlation scenes, a single-scene co-occurrence value of the target factor pair is calculated; Based on the single-scene co-occurrence value, initial co-occurrence frequencies of each factor in the multi-factor matrix in multiple scenes are calculated; The initial co-occurrence frequencies are subjected to frequency standardization processing to obtain scene co-occurrence frequencies. 4.The method of claim 1, wherein, The construction of the three-dimensional correlation matrix of the multi-element animation materials by using the aligned features comprises: The three-order tensor slices of the multi-element animation materials are constructed by using the aligned features; The three-order tensor slices are subjected to high-order singular value decomposition, and the decomposed three-order tensor slices are subjected to singular value compression processing to obtain a low-rank core tensor; The low-rank core tensor is subjected to correlation intensity quantization processing to obtain a correlation core tensor; The three-dimensional correlation matrix of the multi-element animation materials is constructed by using the correlation core tensor.

5. The method of claim 1, wherein the method is based on big data to construct an atlas library of animation materials. The analysis of the frame entropy jump degrees between each image in the image sequences comprises: Each image in the image sequences is converted into a gray-scale image to obtain a set of gray-scale images; The gray-scale histograms of each image in the set of gray-scale images are counted; Single-frame entropies of the images corresponding to the gray-scale histograms are calculated; Based on the single frame entropy, a frame entropy jump degree of adjacent images in the gray image set is calculated.

6. The method of claim 1, wherein the method is implemented based on big data. The three tuple structure text is subjected to animation semantic assignment to obtain a field text, including: The text entity corresponding to the three tuple structure text is subjected to role semantic annotation to obtain an annotated text; The annotated text is subjected to ontology mapping based on the animation field to obtain a subject mapping text; The subject mapping text is subjected to multi-level semantic enhancement processing to obtain an enhanced structure text; The enhanced structure text is subjected to field text reconstruction to obtain a field text. 7.The method of claim 1, wherein, The unstructured audio data is subjected to time-frequency domain processing to extract audio features of the unstructured audio data, including: The unstructured audio data is subjected to pre-emphasis processing to obtain enhanced audio; The enhanced audio is subjected to frame windowing processing to obtain segmented audio; The segmented audio is subjected to short-time Fourier transform to obtain a time-frequency domain spectrum graph; Mel frequency cepstral coefficients of the time-frequency domain spectrum graph are extracted, and the energy, zero-crossing rate and fundamental frequency of each frame of audio in the time-frequency domain spectrum graph are calculated; The mel frequency cepstral coefficients, the energy, the zero-crossing rate and the fundamental frequency are combined to obtain audio features of the unstructured audio data. 8.The method of claim 1, wherein, The audio features, the image features and the text features are subjected to feature alignment to obtain aligned features, including: The audio features, the image features and the text features are subjected to cross-modal vector space projection processing to obtain a feature vector group; The feature vector group is subjected to reinforcement learning to obtain an optimized feature vector; The optimized feature vector is subjected to three tuple dynamic constraint to obtain a boundary separated feature vector; The boundary separated feature vector is subjected to graph attention relationship fusion processing to obtain aligned features. 9.The method of claim 1, wherein, Based on the multi-factor matrix and the correlation weight, a graph library of the multi-element animation material is constructed, including: The multi-factor matrix is used to construct a graph node of the multi-element animation material; Based on the correlation weight, the graph node is linked to obtain a graph grid; The graph grid is used to construct a graph library of the multi-element animation material. 10.A system for constructing an atlas database of animation materials based on big data, characterized in that, The system includes: An image material processing module is configured to collect multi-element animation materials based on big data technology, the multi-element animation materials including image sequences, structured texts and unstructured audio data, analyze frame entropy jump degrees between each image in the image sequences, sample key frames of images in the image sequences based on the frame entropy jump degrees to obtain key frame images, and extract image features of the key frame images; A text material processing module is configured to convert the structured texts into three tuple structure texts, assign animation semantics to the three tuple structure texts to obtain field texts, and extract text features of the field texts; The association matrix construction module is configured to perform time-frequency domain processing on the unstructured audio data to extract audio features of the unstructured audio data, perform feature alignment on the audio features, the image features and the text features to obtain aligned features, and construct a three-dimensional association matrix of the multi-element animation material by using the aligned features. The atlas library construction module is configured to decompose the three-dimensional association matrix into a multi-factor matrix, calculate a scene co-occurrence frequency of each factor in the multi-factor matrix, analyze an association weight of a factor in the multi-factor matrix based on the scene co-occurrence frequency, and construct an atlas library of the multi-element animation material based on the multi-factor matrix and the association weight.

Images