Big data-based material library management system for animation design

Abstract

The invention relates to the technical field of big data processing and cartoon design software, in particular to a cartoon design material library management system based on big data, which comprises a request characterization unit used for determining data load size, delay sensitivity and operation semantic category; the priority arbitration unit is used for calculating a dynamic priority score based on the delay sensitivity, the data load size and the instantaneous occurrence frequency determined by the request characterization unit; the prefetching decision-making unit is used for calculating to obtain a prefetching confidence coefficient and generating a prefetching I / O task based on the prefetching confidence coefficient; the scheduling control unit is used for comparing and analyzing the dynamic priority score calculated by the priority arbitration unit with a preset high priority threshold value and a preset low priority threshold value to generate a tiny differential transmission signal, a large data transmission signal or a mixed transmission signal, and then executing a transmission strategy corresponding to the signal; according to the invention, fair and efficient distribution of I / O resources is realized, and the inherent contradiction between delay and throughput is solved.

Description

Technical Field

[0001] This invention relates to the field of big data processing and animation design software technology, specifically to a big data-based material library management system for animation design. Background Technology

[0002] In animation design work, the system needs to handle a large number of material library access requests. These I / O requests have huge differences in characteristics, including both tiny interactive operations with extremely high response time requirements and large data transmission tasks that require high throughput. Existing resource library management systems typically fail to differentiate the performance requirements of different requests, resulting in an inherent conflict between latency and throughput when handling mixed I / O requests. This makes it easy for designers' real-time interactive operations to be blocked by large data transfers, causing lag and severely impacting workflow smoothness. Furthermore, the system only passively responds to loading requests, failing to preload relevant resources, forcing designers to wait and reducing overall work efficiency. This inability to perceive application-layer semantics and perform differentiated scheduling limits the performance and user experience of animation design software. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a big data-based material library management system for animation design. Specifically, the technical solution of this invention includes: A big data-based resource library management system for animation design includes: The request characterization unit is used to capture atomic operation signaling at the application layer and perform semantic parsing based on the atomic operation signaling to determine the data load size, latency sensitivity, and operation semantic category. The priority arbitration unit is used to calculate a dynamic priority score based on the latency sensitivity, data load size, and instantaneous occurrence frequency determined by the request characterization unit. The prefetch decision unit is used to query the preset asset dependency graph and combine it with the user behavior history when the operation semantic category is the preset asset selection category, calculate the prefetch confidence, and generate a prefetch I / O task based on the prefetch confidence. The scheduling and control unit is used to compare and analyze the dynamic priority score calculated by the priority arbitration unit with the preset high priority threshold and low priority threshold to generate small differential transmission signals, large data transmission signals or mixed transmission signals, and then execute the transmission strategy corresponding to the signal.

[0004] Preferably, the request characterization unit is specifically used for: Parse atomic operation signaling to obtain the data payload size required to process the request; Based on the operation semantic category of the atomic operation signaling, the latency sensitivity is obtained by querying the preset mapping table.

[0005] Preferably, the priority arbitration unit is specifically used for: The data load size is normalized to obtain the normalized data load value; Obtain the instantaneous occurrence frequency of atomic operation signaling, and normalize the instantaneous occurrence frequency to obtain the normalized value of the operation frequency; By combining latency sensitivity, data load normalization, and operation frequency normalization, a dynamic priority score is calculated using a pre-defined heuristic scoring model.

[0006] Preferably, the heuristic scoring model is used to amplify the priority of latency-sensitive requests and suppress the priority of large data volume requests in a non-linear manner; the dynamic priority score is used to reorder the I / O request queue in real time.

[0007] Preferably, the prefetch decision unit is specifically used for: Obtain the correlation strength of related assets in the asset dependency graph; Obtain the frequency of recent accesses by users to related assets; The prefetch confidence score is obtained by weighting the association strength and recent access frequency using a linear weighted model.

[0008] Preferably, the prefetch decision unit is further configured to generate a prefetch I / O task when the prefetch confidence is greater than a preset confidence threshold; and send the generated prefetch I / O task to the request characterization unit for characterization processing and participation in subsequent priority ranking.

[0009] Preferably, the specific processing procedure of the scheduling control unit is as follows: When the dynamic priority score is greater than the high priority threshold, a small differential transmission signal is generated. When the dynamic priority score is less than the low priority threshold, a large block data transmission signal is generated. When the dynamic priority score is between the low priority threshold and the high priority threshold, a mixed transmission signal is generated.

[0010] Preferably, the transmission strategy includes: In response to minute differential transmission signals, a small data packet high-frequency mode is used for transmission; In response to large data transmission signals, a mode of merging requests and large consecutive data packets is used for transmission.

[0011] Compared with the prior art, the present invention has the following beneficial effects: 1. By capturing and parsing the operational signaling of designers at the application layer, this system can accurately identify the semantic intent and performance requirements of different operations. It distinguishes between latency-sensitive real-time interactions and large file transfers that require high throughput, and dynamically assigns differentiated priorities, thereby solving the resource contention and operation lag problems caused by the indiscriminate processing of traditional systems. 2. This system has constructed an intelligent priority arbitration mechanism that comprehensively considers the latency sensitivity of the operation, the size of the data load, and the frequency of instantaneous occurrences to calculate a dynamic score for each request. This mechanism can amplify the priority of high-frequency interactive operations and suppress the priority of large data requests in a non-linear manner, ensuring the smoothness of critical operations and realizing the fair and efficient allocation of I / O resources. 3. By analyzing the inherent relationships between assets and the user behavior history of designers, this system achieves accurate and proactive resource prefetching. The prefetching tasks are seamlessly integrated into a unified priority scheduling system and are always executed using idle bandwidth without interfering with high-priority interactions. This not only shortens the waiting time for designers but also avoids I / O congestion caused by invalid prefetching. 4. This system can translate the logical priority of a request into a specific physical transmission strategy. For high-priority requests, a small data packet high-frequency mode is used to ensure the lowest latency; for low-priority requests, it switches to a large block continuous transmission mode to maximize throughput. This adaptive transmission mode switching solves the inherent contradiction between latency and throughput. Attached Figure Description

[0012] The present invention will be further explained below with reference to the accompanying drawings and embodiments: Figure 1 This is a structural diagram of the system of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0014] Example 1: Please see Figure 1 A big data-based resource library management system for animation design includes: The request characterization unit is used to capture atomic operation signaling at the application layer and perform semantic parsing based on the atomic operation signaling to determine the data load size, latency sensitivity, and operation semantic category. The priority arbitration unit is used to calculate a dynamic priority score based on the latency sensitivity, data load size, and instantaneous occurrence frequency determined by the request characterization unit. The prefetch decision unit is used to query the preset asset dependency graph and combine it with the user behavior history when the operation semantic category is the preset asset selection category, calculate the prefetch confidence, and generate a prefetch I / O task based on the prefetch confidence. The scheduling and control unit is used to compare and analyze the dynamic priority score calculated by the priority arbitration unit with the preset high priority threshold and low priority threshold to generate small differential transmission signals, large data transmission signals or mixed transmission signals, and then execute the transmission strategy corresponding to the signal.

[0015] This embodiment provides a big data-based material library management system for animation design, including a request characterization unit, a priority arbitration unit, a prefetching decision unit, and a scheduling control unit; The purpose of the request characterization unit is to transform the abstract designer operations in the upper-layer application into performance requirements that the underlying I / O system can understand and quantify. In this embodiment, this unit is implemented by deploying a monitor at the application layer of the animation design software. This monitor captures the atomic operation signaling of the designer in real time. The captured signaling is then fed into a semantic parsing model, which can transform unstructured operation signaling into structured feature information containing key I / O performance indicators. This unit performs semantic parsing based on the atomic operation signaling to determine the data load size. Delay sensitivity and operational semantic categories ; The purpose of the priority arbitration unit is to calculate a dynamic priority for each I / O request that fairly reflects its service urgency. In this embodiment, the unit receives feature information output by the request characterization unit and calculates a dynamic priority score for each request based on a heuristic scoring model built to balance real-time interaction and bulk data transfer. This score is the core basis for all subsequent scheduling decisions. It ensures that small operations that are highly sensitive to latency can obtain I / O resources first, and can be responded to in a timely manner even when the system is under pressure from massive data transmission. The purpose of the prefetching decision unit is to transform passive I / O response into proactive resource preloading, thereby reducing the time designers spend waiting for materials to load. In this embodiment, the unit continuously monitors the operation semantic categories parsed by the request characterization unit. When a specific operational semantic category is identified, such as asset selection, the unit initiates a prefetch decision process. This process queries a pre-built asset dependency graph describing the logical relationships between all assets within the project, and combines this with an analysis of the designer's user behavior history to calculate a prefetch confidence level. When the confidence level exceeds a preset threshold, the system will proactively generate a prefetch I / O task for the associated asset. The purpose of the scheduling and control unit is to convert the logical priority calculated by the priority arbitration unit into actual physical I / O transmission behavior. In this embodiment, the unit receives the I / O request queue sorted by priority and assigns the dynamic priority score attached to the request at the head of the queue. Compared with the system's preset high priority threshold and low priority threshold The unit performs comparative analysis. Based on the comparison results, it generates one of the following: a small differential transmission signal, a large data transmission signal, or a mixed transmission signal. It then instructs the underlying I / O scheduler to execute a specific transmission strategy corresponding to the signal, thereby achieving adaptive scheduling and fine-grained control of the data stream. Through the collaborative work of the aforementioned request characterization unit, priority arbitration unit, prefetch decision unit, and scheduling control unit, this invention constructs a complete closed-loop control system from application layer semantic perception to physical layer I / O scheduling. This system can accurately identify the I / O requirements of different design operations, dynamically assign priorities to each request, intelligently prefetch related assets, and ultimately adaptively select the optimal data transmission strategy. This solves the inherent contradiction between latency and throughput in traditional storage systems, greatly improving the smoothness and efficiency of animation design work.

[0016] Example 2: The request characterization unit is specifically used for: Parse atomic operation signaling to obtain the data payload size required to process the request; Based on the operation semantic category of the atomic operation signaling, the latency sensitivity is obtained by querying a preset mapping table; The priority arbitration unit is specifically used for: The data load size is normalized to obtain the normalized data load value; Obtain the instantaneous occurrence frequency of atomic operation signaling, and normalize the instantaneous occurrence frequency to obtain the normalized value of the operation frequency; By combining latency sensitivity, data load normalization value, and operation frequency normalization value, a dynamic priority score is calculated using a pre-defined heuristic scoring model. Heuristic scoring models are used to amplify the priority of latency-sensitive requests and suppress the priority of large data volume requests in a non-linear manner; dynamic priority scores are used to reorder the I / O request queue in real time.

[0017] Based on Embodiment 1, this embodiment further specifies the specific implementation methods of the request characterization unit and the priority arbitration unit; The processing flow of the request characterization unit is more specific; this unit obtains the data load size required to process the request by parsing atomic operation signaling. Data load size δ refers to the amount of data that an I / O request needs to process. Its function is to quantify the weight of the request, and its source is the direct parsing of operation signaling. This unit will look up the latency sensitivity in a preset mapping table according to the operation semantic category of the atomic operation signaling. The delay sensitivity λ is a dimensionless coefficient that quantifies the sensitivity of an operation to response time. It is derived from a mapping table built based on animation design process analysis, which assigns a high value (close to 1.0) to operations that directly affect the artist's visual feedback. The value is assigned to the background operation, which is close to 0. value; The priority arbitration unit calculates the dynamic priority score. The model and steps were precisely defined; to ensure the consistency of physical dimensions and improve the universality of the model, the unit normalized the data load size to obtain the normalized data load value. The normalized data load value δn refers to the dimensionless data load, which aims to eliminate the excessive influence of the absolute value of the data load on priority calculation. Its calculation method is as follows: ,in This unit sets a preset reference data block size for the system; simultaneously, it also acquires the instantaneous occurrence frequency of atomic operation signaling. The instantaneous frequency is then normalized to obtain the normalized value of the operating frequency. The instantaneous frequency φ refers to the number of times a specific operation occurs per unit time. Its function is to identify high-frequency interactive behaviors, and it is derived from real-time statistics collected by the application layer monitor. Its normalization method is... ,in This is a reference frequency, such as the monitor's refresh rate of 60Hz; After normalization, the priority arbitration unit combines latency sensitivity. Normalized data load and normalized value of operation frequency Dynamic priority scores are calculated using a pre-defined heuristic scoring model. The design goal of this heuristic scoring model is to amplify the priority of latency-sensitive requests and suppress the priority of large data volume requests in a non-linear manner. To further clarify, the weighting coefficient The calibration process is as follows: Construct a system containing... The calibration dataset of the test data set, in which the first set of test data is... The set of data includes a preset test I / O request and its corresponding performance metrics. A target priority score pre-set by a domain expert based on the expected response characteristics of the request. Based on this dataset, regression analysis methods such as least squares are used to solve for the result that... Minimized The value serves as the final weighting coefficient, and its specific calculation formula is as follows:

[0018] This formula uses linear superposition of delay sensitivity in molecules. and operating frequency This amplifies the priority of high-frequency, latency-sensitive requests; by using a logarithmic function in the denominator. This achieves non-linear suppression of high-volume data request priorities; the constant 1 is added to the denominator to ensure that when the data load... When the denominator is 0, the denominator is not zero, which ensures the robustness of the model; Calculated dynamic priority score This system is used for real-time reordering of I / O request queues. By introducing normalization processing and a specific heuristic scoring model, the priority calculation becomes more scientific and robust. This model not only quantifies the impact of latency sensitivity but also innovatively incorporates nonlinear suppression of operation frequency and data load, thereby more finely balancing resource competition among different types of I / O requests. The system's QoS arbitration capability is significantly enhanced, enabling more accurate and stable sorting of I / O requests, thus more effectively ensuring the smoothness of high-frequency interactive operations in practical applications.

[0019] Example 3: The prefetch decision unit is specifically used for: Obtain the correlation strength of related assets in the asset dependency graph; Obtain the frequency of recent accesses by users to related assets; The association strength and recent access frequency are weighted by a linear weighting model to obtain the prefetch confidence. The prefetch decision unit is further used to generate a prefetch I / O task when the prefetch confidence is greater than a preset confidence threshold; and to send the generated prefetch I / O task to the request characterization unit for characterization processing and participation in subsequent priority ranking.

[0020] This embodiment, based on embodiment 1, further elaborates on the specific working mechanism and process of the prefetching decision unit; The prefetching decision unit calculates the prefetching confidence level. The model is based on a linear weighted model that combines static asset correlation and dynamic user behavior; this unit needs to obtain the correlation strength of related assets in the asset dependency graph. The asset dependency graph is a pre-built data structure that describes the logical references and dependencies between all assets within a project. The association strength Ga refers to the degree of association between two asset nodes in the graph. Its function is to assess the static logical correlation between assets. It is derived from a dimensionless value obtained by offline calculation and normalization using the graph algorithm. This unit also needs to obtain the user's recent access frequency to related assets. The recent access frequency (Hu) refers to the frequency with which a user accesses a specific asset in a recent period of time. Its purpose is to introduce dynamic user behavior patterns and make prefetching more personalized. It is a dimensionless value obtained by dynamically calculating and normalizing user behavior logs recorded by the system. After obtaining the above two parameters, the prefetching decision unit uses a linear weighted model to assess the correlation strength. and recent visit frequency The prefetch confidence level is obtained by weighting the values. The calculation formula for this model is as follows:

[0021] in and This corresponds to a dimensionless weight, the value of which is adapted to the different production stages of the project; for example, in the creative exploration stage, a specific weight can be set. To enhance the impact of recent user actions on prefetching decisions; in the standardized production phase, settings can be configured... Prioritize following established asset logic relationships; After calculating the prefetch confidence level Subsequently, the prefetch decision unit is further configured to determine if the prefetch confidence level is greater than a preset confidence threshold. When a prefetch I / O task is generated, the confidence threshold θp is a preset trigger threshold. Its function is to filter out low-probability prefetches and avoid invalid prefetches occupying I / O bandwidth. Its source is set based on historical data analysis and system performance testing. After generating a prefetch I / O task, this unit sends the generated prefetch I / O task to the request characterization unit for characterization processing and participation in subsequent priority ranking. This means that the prefetch task does not bypass the system's QoS arbitration mechanism, but is assigned a low-latency-sensitive task. and low instantaneous frequency The feature vectors are queued in the priority arbitration unit along with other ordinary I / O requests. By introducing a linear weighted model that combines static graph association and dynamic user behavior, the accuracy and intelligence of prefetching decisions are greatly improved. By re-sending the generated prefetching tasks to the request feature unit and the priority arbitration queue, this scheme constructs a prefetching closed-loop system, ensuring that prefetching tasks are always executed using the bus's idle bandwidth without interfering with high-priority interactive tasks. This achieves a better loading experience while avoiding I / O congestion that may be caused by traditional prefetching techniques.

[0022] Example 4: The specific processing procedure of the scheduling and control unit is as follows: When the dynamic priority score is greater than the high priority threshold, a small differential transmission signal is generated. When the dynamic priority score is less than the low priority threshold, a large block data transmission signal is generated. When the dynamic priority score is between the low priority threshold and the high priority threshold, a mixed transmission signal is generated; The transmission strategy includes: In response to minute differential transmission signals, a small data packet high-frequency mode is used for transmission; In response to large data transmission signals, a mode of merging requests and large consecutive data packets is used for transmission.

[0023] Based on Example 1, this embodiment provides a detailed definition of the decision-making logic of the scheduling control unit and the specific transmission strategies it triggers; The specific processing procedure of the scheduling control unit is a rule set based on dual thresholds, which aims to map quantified priority scores to explicit physical transmission behaviors; its decision rules are as follows: When dynamic priority score Greater than the high priority threshold At that time, a tiny differential transmission signal is generated; the high priority threshold θh is a preset dimensionless threshold, which is used to identify those requests that are extremely sensitive to latency and require immediate response. It is determined based on the system's latency and throughput performance calibration test under different loads. When dynamic priority score Less than the low priority threshold At that time, a large block data transmission signal is generated; the low priority threshold θl is another preset dimensionless threshold, which is used to identify large block data tasks that have high throughput requirements but are not sensitive to latency. When dynamic priority score Between low priority thresholds With high priority threshold When in between, a mixed transmission signal is generated; The transmission strategy is the specific physical transmission mode triggered by the aforementioned signal, specifically including: In response to tiny differential transmission signals, the I / O scheduler uses a small data packet, high-frequency transmission mode. In this mode, the system disables algorithms that may introduce delays to ensure that data packets can be sent immediately, thereby achieving the lowest possible interaction latency. In response to large data transfer signals, the I / O scheduler will use a mode of merging requests with large, consecutive data packets for transmission; this mode is designed to maximize bus utilization and data throughput. In response to mixed transmission signals, the system can adopt a balancing strategy that lies between the two; By defining a clear dual-threshold decision logic and corresponding specific transmission strategies, the concept of priority is successfully transformed into an executable physical layer operation. Instead of using a single scheduling algorithm, this scheme adaptively switches between latency-preserving and throughput-preserving modes based on the dynamic priority of requests, thereby achieving refined and differentiated use of physical I / O resources.

[0024] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A material library management system for animation design based on big data, characterized by, The application comprises the following steps: The request characterization unit is used to capture atomic operation signaling of the application layer and perform semantic analysis based on the atomic operation signaling to determine data load size, delay sensitivity and operation semantic category; The priority arbitration unit is used to calculate a dynamic priority score based on the delay sensitivity, data load size and instantaneous occurrence frequency determined by the request characterization unit; The prefetch decision unit is used to query a preset asset dependency graph and combine user behavior history to calculate a prefetch confidence when the operation semantic category is a preset asset selected category, and generate a prefetch I / O task based on the prefetch confidence; The scheduling control unit is used to compare and analyze the dynamic priority score calculated by the priority arbitration unit with a preset high priority threshold and a preset low priority threshold to generate a small differential transmission signal, a large data transmission signal or a mixed transmission signal, and then execute a transmission strategy corresponding to the signal.

2. The material library management system for animation design based on big data according to claim 1, wherein The request characterization unit is specifically used to: analyze the atomic operation signaling to obtain data load size required for processing by the request; query a preset mapping table according to the operation semantic category of the atomic operation signaling to obtain delay sensitivity.

3. The material library management system for animation design based on big data according to claim 2, wherein The priority arbitration unit is specifically used to: normalize the data load size to obtain a data load normalized value; obtain the instantaneous occurrence frequency of the atomic operation signaling and normalize the instantaneous occurrence frequency to obtain an operation frequency normalized value; combine the delay sensitivity, data load normalized value and operation frequency normalized value to calculate a dynamic priority score through a preset heuristic scoring model.

4. The material library management system for animation design based on big data according to claim 3, wherein The heuristic scoring model is used to amplify the priority of delay-sensitive requests and suppress the priority of large data volume requests in a non-linear manner; the dynamic priority score is used to real-time reorder an I / O request queue.

5. The material library management system for animation design based on big data according to claim 1, wherein The prefetch decision unit is specifically used to: obtain the correlation strength of the associated asset in the asset dependency graph; obtain the recent access frequency of the associated asset by the user; weight the correlation strength and recent access frequency through a linear weighting model to obtain a prefetch confidence.

6. The material library management system for animation design based on big data according to claim 5, wherein The prefetch decision unit is further used to generate a prefetch I / O task when the prefetch confidence is greater than a preset confidence threshold; and send the generated prefetch I / O task to the request characterization unit for characterization processing and participation in subsequent priority sorting.

7. The material library management system for animation design based on big data according to claim 1, wherein The specific processing process of the scheduling control unit is as follows: generate a small differential transmission signal when the dynamic priority score is greater than the high priority threshold; generate a large data transmission signal when the dynamic priority score is less than the low priority threshold; generate a mixed transmission signal when the dynamic priority score is between the low priority threshold and the high priority threshold.

8. The material library management system for animation design based on big data according to claim 7, wherein The transmission strategy comprises: responding to the small differential transmission signal, adopt a small data packet high frequency mode for transmission; responding to the large data transmission signal, adopt a mode of merging requests and large continuous data packets for transmission.

Images