Construction method of role IP enabling linkage control model

Abstract

The invention discloses a construction method and system of a role IP enabling linkage control model, the system comprises a sensor array module, a GPU-CPU heterogeneous computing system and an infrared eye movement tracking module as a physical feedback module, and the sensor array module collects multi-modal biological and behavior data from a performer through a fixed physical layout; according to the heterogeneous computing system, through hardware algorithm distribution, a GPU is used for executing a geometric deep learning network, and a CPU is used for executing a recurrent neural network so as to efficiently process data and fuse the data into a unified role IP state vector; and the physical feedback module carries out closed-loop online optimization on a state vector generation strategy by capturing eye movement data of a terminal observer as a reward signal of a reinforcement learning algorithm. Through a specific hardware closed-loop system, the adaptive cost is greatly reduced, and meanwhile, the IP expressive force and the audience immersion degree generate nonlinear technical effect jump.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of digital content creation and artificial intelligence, and specifically relates to a role IP enabled linkage control model construction method and system, more specifically, the present application relates to the use of multi-modal sensor fusion, heterogeneous computing and closed-loop feedback control technology to realize real-time, adaptive animation generation of digital characters. BACKGROUND

[0002] In the current field of digital content creation, especially in the game development, film and television animation and virtual reality production industry, a universal technical axiom has been followed for a long time: there are fundamental differences in rendering pipelines and asset standards between different platforms, therefore, cross-platform reuse of the same IP asset must follow the core process of platform priority-manual adaptation, which requires the technical and artistic team to manually simplify the model, redraw the map, reset the bone binding and reprogram the animation state machine for each target platform's characteristics, such as from offline rendering to real-time rendering in game engine.

[0003] However, the present inventors have found that strict adherence to the above-mentioned manual adaptation axiom has led to a cost-performance-security paradox that current technology cannot reconcile: First, in terms of cost, each additional application platform for the same IP asset requires a complete, high-skilled team to invest several weeks or even months in manual translation and adaptation, which increases the total cost of IP development throughout its life cycle by at least 60%-80%, greatly wasting creative capacity.

[0004] Secondly, in terms of performance and fidelity, manual adaptation is essentially a lossy replication process, and the core settings of the character, such as micro-expression, specific dynamic interaction logic, etc., inevitably lose semantic and expressive power during the translation process, which leads to different performances of the same character experienced by users in different works, seriously weakening the coherence and immersion of the IP, and forming a negative effect of the more platforms-the more fragmented IP.

[0005] Finally, in terms of intellectual property security, core digital assets are manually transferred and reproduced between multiple teams and multiple platforms, greatly increasing the risk nodes of data leakage and piracy infringement, and each time the manual adaptation is increased, a copy of the core asset is added, leading to a risk of uncontrolled management that is multiplied, posing a serious threat to the security of IP assets.

[0006] Therefore, the technical problem to be solved urgently by the prior art is: how to break the technical axiom of platform priority-manual adaptation, eliminate the inherent technical defects such as high cost, IP experience fragmentation and data security risk brought by the above-mentioned platform priority-manual adaptation, and the technical solutions disclosed in the prior art, such as Chinese patents CN111798552A and CN111462284A, cannot fundamentally solve this paradox. SUMMARY

[0007] The present application aims to provide a role IP enabling linkage control model construction method and system, which aims to solve the technical problems of high cost, IP experience fragmentation and data security risk caused by following the platform priority-manual adaptation principle in the prior art. The present application provides a role IP enabling linkage control system and the corresponding model construction method, which no longer relies on offline, lossy manual adaptation process, but constructs a hardware closed-loop system based on multi-modal biological feature perception, heterogeneous computing processing and human factors engineering feedback.

[0008] To achieve the above-mentioned purpose, the present application adopts the following technical solutions, and the system specifically includes the following three core modules: The sensor array module synchronously collects rich multi-modal data sets from the performer through a fixed physical structure, including three-dimensional facial geometry data, spatial audio data and physiological signal data, which guarantees the integrity and consistency of the data from the source.

[0009] The heterogeneous computing system uses a hardware-software collaborative design to allocate different types of data to the optimal processor and algorithm for processing, wherein the optimal processor is specifically GPU for processing three-dimensional geometry data and CPU for processing timing data, and the algorithm is specifically a geometric deep learning network and a recurrent neural network algorithm to extract features with maximum fidelity. Subsequently, these features are integrated into a single, platform-independent role IP state vector through a probabilistic fusion model.

[0010] The physical closed-loop feedback module based on eye tracking is a key innovation point of the present application. This module captures the implicit, subconscious physiological reactions of the end user, i.e. the audience, and uses these reaction data as real-time reward signals to continuously optimize the aforementioned fusion model within a reinforcement learning framework. This ensures that the IP's performance is not only technically accurate, but also emotionally resonates with the audience to the greatest extent and interacts with them.

[0011] Compared with the prior art, the present application has the following beneficial effects: The application realizes the nonlinear technical effect leap by the specific technical scheme, especially by introducing the hardware closed-loop feedback mechanism, which not only reduces the cross-platform adaptation cost to almost zero, ensures the fidelity of IP emotional expression, and significantly improves the effective participation and immersion of the audience. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiments will be briefly introduced.

[0013] Figure 1 The hardware structure and data flow diagram of the role IP enabled linkage control system of the application; Figure 2 The flowchart of the construction and optimization method of the role IP enabled linkage control model of the application. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0015] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application. Figure 1 and Figure 2 The specific embodiments of the application will be described in detail below, so that those skilled in the art can more fully understand the technical solutions of the application.

[0016] With reference to Figure 1 The overall architecture of the role IP enabled linkage control system provided by the embodiments of the application includes a sensor array module 1, a heterogeneous computing system 2 and a physical closed-loop feedback module 3.

[0017] The sensor array module 1 is specifically a multi-modal input stage. The design principle of this module is that the comprehensive understanding of human emotional state needs to integrate data from multiple information sources. The application combines autonomous motion expression and non-autonomous physiological arousal. The autonomous motion expression can be facial or voice, and the non-autonomous physiological arousal can be skin conductance response. Because the physiological signal is not easy to fake, it can provide a more reliable benchmark for emotional state. This multi-modal method ensures that the input signal is more rich and robust than any single mode. Specifically, the module 1 includes: The three-dimensional geometry data acquisition unit 101 adopts a high-precision three-dimensional scanner to capture dense facial point cloud data, and specifically can adopt a structured light or time-of-flight sensor, which is crucial because three-dimensional geometry can capture subtle contours and micro-expressions that are essential for emotional fidelity, which are lost in two-dimensional image projections.

[0018] The audio acquisition unit 102 adopts a pair of MEMS microphones and is arranged in a specific stereo configuration, which can be symmetrically distributed around the optical center of the unit 101 with a distance of 150±2mm. This precise physical layout is not only used to capture language content, but more importantly, it is used to capture paralinguistic cues such as pitch, intonation, and rhythm, which are crucial for speech emotion recognition.

[0019] The physiological signal acquisition unit 103 adopts a galvanic skin response (GSR) sensor to measure changes in skin conductivity, which provides a direct and quantitative measure of autonomic nervous system arousal levels, which is a key component of emotional response.

[0020] The present application innovates in the fixed physical layout of the sensor array, which is not a simple stacking of sensors, but rather a solution to the core technical challenges in multi-modal fusion. Firstly, it ensures strict time synchronization between data streams, which is a major difficulty in multi-modal fusion analysis. Secondly, it provides a unified spatial reference framework for visual data such as three-dimensional scanning and auditory data such as microphones, which is important for precise lip synchronization and sound source localization. This physically integrated design turns a group of discrete sensors into a single, cohesive biometric capture device, which is fundamentally different from using independent devices in a temporary combination.

[0021] The heterogeneous computing system 2 is a specialized processing and fusion level. The system 2 adopts a hardware-software collaborative design concept. Different types of data have fundamentally different structures and require specialized processing methods. A general processing scheme would sacrifice speed and accuracy. Therefore, the system assigns each task to the optimal computing substrate.

[0022] GPU-based geometry processing. The three-dimensional facial point cloud data stream obtained from the three-dimensional geometry data acquisition unit 101 is directly sent to the GPU, which executes a geometric deep learning network (GDN). Numerous studies have shown that GDN is the latest technology for analyzing three-dimensional geometric data such as point clouds and meshes, which can effectively learn features related to facial shape and expression while decoupling these features from identity information.

[0023] Based on CPU timing processing, one-dimensional time series data from the audio unit 102 and the GSR sensor 103 is sent to the CPU, and a long short-term memory (LSTM) network, which is a kind of recurrent neural network, is executed on the CPU, researches have confirmed that LSTM performs well in modeling long-term dependencies in sequential data, making it very effective in speech emotion recognition and physiological signal analysis.

[0024] Based on Bayesian model-based feature fusion, high-order feature vectors from GDN and LSTM, where GDN is on GPU and LSTM is on CPU, are integrated by CPU using a Bayesian fusion model, the Bayesian framework is selected because it can handle uncertainty in a probabilistic way and combine evidence from different sources, this model learns how to weight the contributions of different modalities according to the context, for example, in a scene that requires subtle emotional expression, the model may learn to give greater weight to GSR and micro-expression data, while in a dialogue-intensive scene, it may prioritize audio and lip-sync data, this fusion model plays the role of an intelligent arbitrator, ultimately generating a character IP state vector that represents the most optimal explanation of the actor's complete emotional and physical state.

[0025] The physical closed-loop feedback module 3 is specifically the innovative core of real-time optimization, and the final measure of IP performance is its impact on the audience, the prior art system operates in an open-loop mode - that is, performance is generated and displayed, but there is no mechanism to adjust according to the audience's response.

[0026] The present application solves this fundamental defect through closed-loop design, the specific working mechanism is that the infrared eye tracking unit is integrated into the display device of the terminal user such as VR headsets or displays, it captures implicit, non-verbal feedback signals, such as pupil diameter related to arousal / cognitive load and fixation points related to attention / confusion, which provides a series of objective, real-time data streams about user engagement.

[0027] Direct application of the RLUF concept sets up reinforcement learning based on user feedback (RLUF), the system defines a reward function based on eye tracking data, for example, sustained fixation on the character's face is a positive reward, while gaze avoidance or irregularities indicate confusion and scanning constitute negative rewards, IP performance distortion events are a specific implementation of this reward function, the Q-learning algorithm running on the CPU or other suitable reinforcement learning agent uses this reward signal to optimize the parameters of the Bayesian fusion model online, where the parameters are the fusion weights of each modality.

[0028] This feedback loop transforms the system from a content generation tool into a content co-creation system, the IP's performance is no longer static or entirely determined by the performer; it is dynamically and continuously adjusted to maximize the audience's subconscious engagement, the system is able to learn in real-time what performance style is effective for a particular user or audience group, for example, if the audience's gaze data indicates that they missed a subtle emotional cue, the reinforcement learning agent can adjust the fusion weights to slightly amplify the features corresponding to that expression in the state vector, this creates a symbiotic relationship between the digital character and the audience, a concept that goes far beyond traditional animation production processes.

[0029] Please refer to Figure 2 The application also provides a method for constructing an IP-enabled linkage control model of a character, and the specific steps of the method correspond to the functions of the modules of the system one by one. Step S1, data acquisition, start the sensor array module 1, and synchronously acquire the facial point cloud, spatial audio and GSR data of the performer.

[0030] Step S2, feature extraction, the heterogeneous computing system 2 processes the data in parallel, the GDN on the GPU processes the point cloud, and the LSTM on the CPU processes the audio and GSR signals, and respectively outputs a geometric feature vector and a temporal feature vector.

[0031] Step S3, feature fusion, the Bayesian fusion model on the CPU integrates the outputs of step S2 to generate an initial character IP state vector, which can be directly used by the downstream rendering engine to drive the character model of any platform.

[0032] Step S4, feedback acquisition, the physical closed-loop feedback module 3 monitors the eye movement data of the terminal observer in real time, including the gaze heat map and the change in pupil diameter.

[0033] Step S5, closed-loop optimization, the system calculates a reward signal according to the data collected in step S4, for example, an event in which the instantaneous change rate of the pupil diameter exceeds a preset threshold or the gaze point deviates from the core area of the character's face for a long time is determined as an IP performance distortion event, and the frequency of the event is taken as a loss function or a negative reward, a learning algorithm such as Q-learning adjusts the weight parameters of the Bayesian fusion model on the CPU according to this signal, and this step is a continuous cycle, so that the model can adaptively adjust the IP performance to maximize the audience's immersion.

[0034] With the above specific technical solutions, especially the introduction of a hardware-based closed-loop control path based on the implicit physiological feedback of the terminal observer, the application produces unexpected technical effects compared to the prior art and its simple combination. In order to verify these effects, strict comparative experiments were conducted, and the results are shown in the following table. Table 1: Comparison of core performance indicators of different technical solutions The beneficial effects of the present application are revealed by analyzing the above experimental data: The advantage of fundamental cost reduction, as shown in the table, the experimental group of the present application reduces the new platform adaptation cost from 48 person-hours / platform of the prior art control group 1 to almost zero 0.5 person-hours / platform, which proves that the platform-independent role IP state vector scheme of the present application fundamentally eliminates the manual adaptation process and realizes a qualitative leap.

[0035] The advantage of significant improvement in fidelity, the emotional fidelity score of the present application reaches 96.8, which is significantly better than the 61.5 of the prior art and the 85.3 of the simple open-loop combination, proving the key role of closed-loop optimization in improving IP expressiveness.

[0036] The advantage of unexpected synergistic gain effect, the most core innovation point of the present application is reflected in the audience effective participation index. Compared with the control group 1, the simple combination of the control group 2 has been significantly improved, proving the value of multi-modal input and heterogeneous computing.

[0037] However, the present application of the experimental group on the basis of the control group 2, this index produces a nearly 75% nonlinear technical effect leap, that is, from 25.8 seconds to 45.2 seconds. This disproportionate huge gain cannot be foreseen by simply adding an eye tracker to the animation system, and it proves that the specific hardware closed-loop system architecture proposed by the present application, that is, using implicit physiological feedback as a reward signal for reinforcement learning to optimize the multi-modal fusion model in real time, produces unexpected synergistic gain, which shows that the present application is not a simple superposition of the prior art, but an organic whole that produces new functions and performance.

[0038] In summary, the present application successfully solves the fundamental problem of role IP cross-platform application in the prior art through its unique system architecture and method, bringing significant economic benefits and technical advantages.

[0039] The above is only the preferred embodiment of the present application, the protection scope of the present application is not limited to the above-mentioned embodiments, any technical solution falling within the scope of the present application is within the protection scope of the present application. It should be noted that for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application are also considered to be within the protection scope of the present application.

Claims

1. A system for constructing a character IP-enabled linkage control model, characterized in that, include: A sensor array module (1), configured to acquire multimodal data from a human body, the module comprising fixed physical components integrating: Three-dimensional geometric data acquisition unit (101) for capturing facial point cloud data; At least two spatially separated audio acquisition units (102) for capturing spatial audio data. And a physiological signal acquisition unit (103) for capturing skin conductance response (GSR) data; wherein the physical layout of the unit is predetermined to ensure the temporal synchronization and spatial consistency of the acquired multimodal data; The heterogeneous computing system (2) is communicatively coupled to the sensor array module (1), and the system includes a graphics processing unit (GPU) and a central processing unit (CPU). The GPU is specifically configured to execute a geometric deep learning network to extract high-order geometric features from the facial point cloud data; the CPU is specifically configured to execute a recurrent neural network to extract temporal features from the spatial audio data and the GSR data. Furthermore, the CPU is configured to execute a probabilistic fusion model to integrate the geometric features and the temporal features into a unified, platform-independent role-IP state vector; A physical closed-loop feedback module (3) is communicatively coupled to the heterogeneous computing system (2), the module including an eye-tracking unit configured to be integrated with a terminal display device; The eye-tracking unit is configured to collect implicit feedback data from the terminal observer, and the implicit feedback data includes at least pupil diameter changes and fixation point distribution. Furthermore, the heterogeneous computing system (2) is also configured to utilize the implicit feedback data as a reward signal in the reinforcement learning algorithm to perform online optimization of the parameters within the probabilistic fusion model.

2. A method for constructing a character IP-enabled linkage control model, characterized in that, Includes the following steps: Step S1: Collect time-synchronized multimodal data streams from a human subject using a sensor array module with a predetermined physical layout. The data streams include facial point cloud data, spatial audio data, and skin conductance response data. Step S2: Within the heterogeneous computing system, the facial point cloud data is processed using a geometric deep learning network executed on the GPU to obtain geometric features, and the spatial audio data and the skin conductance response data are processed using a recurrent neural network executed on the CPU to obtain temporal features. Step S3: By fusing the geometric features and the temporal features through a probabilistic fusion model executed on the CPU, a unified, platform-independent role IP state vector is generated; Step S4: Implicit feedback data is collected from the terminal observer through an eye-tracking unit integrated with the terminal display device, the data including at least pupil diameter and fixation point; Step S5: By using a reinforcement learning algorithm and the reward signal derived from the implicit feedback data, the parameters of the probabilistic fusion model are continuously optimized to form a closed-loop control system.

3. The method for constructing a character IP-enabled linkage control model according to claim 2, characterized in that, The geometric deep learning network is adapted for 3D point cloud analysis.

4. The method for constructing a character IP-enabled linkage control model according to claim 2, characterized in that, The recurrent neural network is a Long Short-Term Memory (LSTM) network.

5. The method for constructing a character IP-enabled linkage control model according to claim 2, characterized in that, The probabilistic fusion model is a Bayesian fusion model.

6. The method for constructing a character IP-enabled linkage control model according to claim 2, characterized in that, The reinforcement learning algorithm is the Q-learning algorithm.

7. The method for constructing a character IP-enabled linkage control model according to claim 2, characterized in that, The reward signal is calculated by quantifying IP performance distortion events, which are defined as the terminal observer's gaze point deviating from the core area of ​​the character's face for a long time, or the instantaneous rate of change of the pupil diameter exceeding a preset threshold.

8. The system for constructing a character IP-enabled linkage control model according to claim 1, characterized in that, In the physical layout of the sensor array module (101), the two audio acquisition units (102) are arranged in a horizontally symmetrical manner with the optical center of the three-dimensional geometric data acquisition unit (101) as the center, and their physical distance is set to 150±2mm.

9. The method for constructing a character IP-enabled linkage control model according to claim 2, characterized in that, The character IP state vector includes decoupling parameters for facial expressions, lip-sync, and emotional intensity.

10. The method for constructing a character IP-enabled linkage control model according to claim 2, characterized in that, The online optimization adjusts the weighting coefficients of the Bayesian fusion model, thereby dynamically changing the influence of different input modalities on the final role IP state vector.

Images