Method and device for controlling in-vehicle virtual object in vehicle, vehicle and storage medium

Abstract

The embodiment of the application provides a kind of control method, device, vehicle and storage medium of virtual object in vehicle in vehicle, the method comprises: obtaining the state perception data of driver in vehicle, and the travel data of vehicle, wherein, state perception data is used to represent the emotion of driver, travel data is used to represent the travel state of vehicle;State perception data and travel data are fused to obtain fusion data;Determine the control instruction of virtual object to be displayed in vehicle based on fusion data in vehicle, wherein, control instruction is used to represent the display rule of the bone action of virtual object in vehicle;Obtain a plurality of bone animations matched with control instruction;According to a plurality of bone animations and control instruction, the bone action of virtual object in vehicle is controlled.The application solves the technical problem that virtual object generation is low in efficiency.

Description

Technical Field

[0001] This application relates to the field of vehicle human-machine interaction technology, and more specifically, to a control method, device, vehicle, and storage medium for in-vehicle virtual objects. Background Technology

[0002] Currently, as intelligent car cockpits develop towards emotionalization, in-vehicle virtual objects are gradually becoming an important human-computer interaction medium in vehicle systems.

[0003] In related technologies, the methods for generating motion in in-vehicle virtual objects can include: based on preset animations and state machines, and based on real-time graphics rendering. Systems based on preset animations and state machines typically require storing a large number of animation files, such as sequence frames, skeletal animation clips, or video files, and manage different interaction scenarios through state machines. Whenever a command is issued or a specific event is triggered, the corresponding preset animation clip is invoked and played according to pre-defined logic. Real-time graphics rendering typically generates high-quality, fully computer-generated virtual character motion through advanced graphics processing and rendering algorithms. However, the above methods suffer from the technical problem of low efficiency in virtual object generation.

[0004] There is currently no good solution to the above problems. Summary of the Invention

[0005] This application provides a method, apparatus, vehicle, and storage medium for controlling virtual objects in a vehicle, in order to at least solve the technical problem of low efficiency in generating virtual objects.

[0006] According to one aspect of the embodiments of this application, a control method for an in-vehicle virtual object is provided. The method may include: acquiring state perception data of the driver in the vehicle and driving data of the vehicle, wherein the state perception data is used to represent the driver's emotions and the driving data is used to represent the driving state of the vehicle; fusing the state perception data and the driving data to obtain fused data; determining control instructions for an in-vehicle virtual object to be displayed in the vehicle based on the fused data, wherein the control instructions are used to represent the display rules of the skeletal movements of the in-vehicle virtual object; acquiring multiple skeletal animations matching the control instructions; and controlling the skeletal movements of the in-vehicle virtual object according to the multiple skeletal animations and the control instructions.

[0007] Furthermore, the method may also include: acquiring multiple interaction commands of the vehicle at multiple consecutive moments, wherein the multiple interaction commands are used to characterize the changing trend of the driver's driving intention; and fusing state perception data and driving data to obtain fused data, including: fusing state perception data, driving data, and interaction commands to obtain fused data.

[0008] Furthermore, the state perception data, driving data, and interaction commands are fused to obtain fused data, including: aligning the state perception data, driving data, and interaction commands with timestamps; encoding the time-stamp-aligned state perception data, driving data, and interaction commands respectively to obtain first encoded data corresponding to the state perception data, second encoded data corresponding to the driving data, and third encoded data corresponding to the interaction commands; and fusing the first encoded data, second encoded data, and third encoded data to obtain fused data.

[0009] Furthermore, acquiring multiple skeletal animations that match the control instructions includes: determining multiple skeletal animations that match the action sequence from the basic action library according to the action sequence in the control instructions, wherein the action sequence includes multiple sub-actions, and the sub-actions correspond to the skeletal animations; and acquiring the multiple skeletal animations sequentially according to the storage order of the multiple sub-actions.

[0010] Furthermore, the skeletal movements of the vehicle-mounted virtual object are controlled according to multiple skeletal animations and control commands, including: scaling the skeletal animations on the time axis according to the speed adjustment coefficients corresponding to the skeletal animations in the control commands, wherein the speed adjustment coefficients are used to adjust the playback speed of the skeletal animations; performing smooth blending processing on the scaled multiple skeletal animations according to the fusion weight parameters in the control commands to obtain a composite skeletal animation trajectory, wherein the fusion weight parameters are used to characterize the superposition ratio of the multiple skeletal animations; and controlling the skeletal movements according to the composite skeletal animation trajectory.

[0011] Furthermore, the method may also include: performing inertial correction on the skeletal motion according to driving data; and adjusting the skeletal motion according to the changed state perception data in response to a change in state perception data.

[0012] Furthermore, based on the fused data, the control commands for the in-vehicle virtual objects to be displayed in the vehicle are determined, including: mapping the fused data to obtain an action sequence; identifying the fused data to determine the driver's emotions and the vehicle's driving state; using the emotions and driving state to determine the speed adjustment coefficient matching the action sequence, and using the emotions to determine the fusion weight parameters matching the action sequence; combining the action sequence, the speed adjustment coefficient matching the action sequence, and the fusion weight parameters matching the action sequence to obtain the control commands.

[0013] According to another aspect of the embodiments of this application, a control device for an in-vehicle virtual object is also provided. The device may include: a first acquisition unit, configured to acquire state perception data of the driver in the vehicle and driving data of the vehicle, wherein the state perception data is used to characterize the driver's emotions and the driving data is used to characterize the driving state of the vehicle; a fusion unit, configured to fuse the state perception data and the driving data to obtain fused data; a determination unit, configured to determine control instructions for an in-vehicle virtual object to be displayed in the vehicle based on the fused data, wherein the control instructions are used to characterize the display rules of the skeletal movements of the in-vehicle virtual object; a second acquisition unit, configured to acquire multiple skeletal animations matching the control instructions; and a control unit, configured to control the skeletal movements of the in-vehicle virtual object according to the multiple skeletal animations and the control instructions.

[0014] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods of various embodiments of this application when it runs.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to run the methods of various embodiments of this application.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that implements the methods of various embodiments of this application when run by a processor.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which is executed by a processor to implement the methods in various embodiments of this application.

[0018] According to another aspect of the embodiments of this application, a computer program is also provided, which is executed by a processor to implement the methods of the various embodiments of this application.

[0019] In this embodiment, driver state perception data and vehicle driving data are acquired. The state perception data represents the driver's emotions, and the driving data represents the vehicle's driving state. The state perception data and driving data are fused to obtain fused data. Based on the fused data, control instructions for the in-vehicle virtual object to be displayed are determined. These control instructions represent the display rules for the skeletal movements of the in-vehicle virtual object. Multiple skeletal animations matching the control instructions are acquired. The skeletal movements of the in-vehicle virtual object are controlled according to the multiple skeletal animations and the control instructions. In other words, in this application, control instructions for the in-vehicle virtual object are determined based on the driver's state perception data and the vehicle driving data. These control instructions can be used to determine the display rules for the skeletal movements of the in-vehicle virtual object, such as display time and display speed. Furthermore, the skeletal animations matching the control instructions can be invoked, and the skeletal movements of the in-vehicle virtual object can be adjusted according to the skeletal animations and the control instructions, thereby achieving the goal of accurately displaying the virtual object. This improves the efficiency of virtual object generation and solves the technical problem of low virtual object generation efficiency. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a flowchart of a method for controlling an in-vehicle virtual object according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of a method for generating in-vehicle virtual avatar actions based on an AI action strategy network, according to an embodiment of this application.

[0023] Figure 3 This is a flowchart of an in-vehicle virtual avatar interaction method based on an action policy network according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of a control device for an in-vehicle virtual object according to an embodiment of this application;

[0025] Figure 5 This is a structural block diagram of a computer terminal according to an embodiment of the present invention;

[0026] Figure 6 This is a block diagram of an electronic device for a method of controlling an in-vehicle virtual object in a vehicle according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] According to an embodiment of this application, a method embodiment for controlling an in-vehicle virtual object in a vehicle is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] Currently, as intelligent car cockpits develop towards emotionalization, in-vehicle virtual objects are gradually becoming an important human-computer interaction medium in vehicle systems.

[0031] In related technologies, the generation of virtual object actions can be categorized into two methods: based on preset animations and state machines, and based on real-time graphics rendering. Systems based on preset animations and state machines typically require storing a large number of animation files, such as sequence frames, skeletal animation clips, or video files, and manage different interaction scenarios through state machines. Whenever a user issues a command or triggers a specific event, the corresponding preset animation clip is invoked and played according to pre-defined logic. Real-time graphics rendering, on the other hand, typically uses advanced graphics processing and rendering algorithms to generate high-quality, fully computer-generated virtual character actions in real time.

[0032] However, the motion-driven approach based on preset animations and state machines, which involves pre-creating a large number of animation files and using state machines to switch between different interactive states to display the virtual avatar's movements, has the following problems: the movements are highly dependent on pre-made resources, making it difficult to adapt to subtle changes during the interaction process, resulting in stiff and repetitive movements; to cover diverse interaction scenarios, a large number of animation resources need to be stored, occupying a significant amount of vehicle system storage space; and the movement switching depends on state transitions, which can easily lead to interrupted or abrupt transitions when user commands are interrupted or the environment changes, resulting in a lack of continuity. While virtual avatar generation technology based on real-time graphics rendering offers some flexibility, it requires high computing power and power consumption in in-vehicle applications, which is not conducive to the long-term stable operation of the vehicle system and makes it difficult to balance movement quality and real-time system response under limited hardware resources.

[0033] Meanwhile, the preset animations are fixed sequences created offline, unable to dynamically adjust to subtle changes during real-time interaction, such as changes in user speech rate, command interruptions, or vehicle posture changes. When a new interaction is triggered during animation playback, the current animation needs to be forcibly interrupted and switched to another, resulting in abrupt changes in the virtual character's movements and affecting the naturalness of the interaction. The animation resources are massive, putting a heavy burden on the vehicle's storage. To improve the flexibility of movements, some related technologies have attempted to use artificial intelligence models to directly generate the skeletal posture or action sequences of the virtual character in real time. This method has high requirements for computing power and power consumption in in-vehicle scenarios and is not suitable for long-term edge operation. Real-time generation of skeletal posture or animation frames usually requires large-scale neural network models for continuous inference, placing high demands on the vehicle's computing resources (e.g., CPU / GPU) and power consumption, which can easily affect the stability of critical functions such as navigation and driver assistance.

[0034] In summary, existing technologies have not yet solved the problem of achieving high flexibility, high continuity, and high realism of in-vehicle virtual avatar movements under limited computing and storage resources on the in-vehicle side, while also ensuring system stability and maintainability. This results in low virtual object generation efficiency. To address these issues, this application proposes an action generation method centered on an action policy network. This method determines control commands for the in-vehicle virtual object based on driver state perception data and vehicle driving data. It then calls upon skeletal animations matching the control commands and adjusts the skeletal movements of the in-vehicle virtual object according to the skeletal animations, thereby achieving accurate display of the virtual object. By outputting high-level action control parameters instead of directly generating low-level animation data, the contradiction between resource consumption and action performance can be resolved to some extent, thus improving the efficiency of virtual object generation and solving the problem of low virtual object generation efficiency.

[0035] This embodiment provides a method for controlling in-vehicle virtual objects in a vehicle. Figure 1This is a flowchart illustrating a method for controlling an in-vehicle virtual object according to an embodiment of this application. Figure 1 As shown, the method may include the following steps.

[0036] Step S102: Obtain the driver's state perception data and the vehicle's driving data, wherein the state perception data is used to characterize the driver's emotions and the driving data is used to characterize the vehicle's driving state.

[0037] In the technical solution provided in step S102 of this application, the aforementioned state perception data can be user emotion data, which can be used to characterize the driver's emotions. This data may include, but is not limited to: image data collected by a camera, which can be used to identify the driver's facial expressions; gaze direction obtained through an eye-tracking system; and voice tone and speech rate features analyzed through speech recognition. The aforementioned facial expressions may include, but are not limited to: the degree of upward turn of the corners of the mouth, the degree of eyelid opening and closing, and the degree of brow crease. The aforementioned gaze direction may include, but is not limited to, the pitch and yaw angles of the gaze in the three-dimensional coordinate system of the vehicle screen. The aforementioned voice tone and speech rate features may include, but are not limited to, the fundamental frequency mean, the rate of change of speech rate, and the frequency of pauses.

[0038] Optionally, image and voice data related to the driver can be collected through devices such as cameras and microphones in the vehicle. By recognizing the image and voice data, driver state perception data can be obtained, which can be used to characterize the driver's emotions.

[0039] Optionally, state perception data can be obtained by collecting the biometrics of drivers and passengers. For example, facial expressions of the driver can be identified in real time through cameras, such as smiling, frowning, and fatigue. The driver's gaze direction can be analyzed through an eye-tracking system to determine the driver's level of attention, such as whether they are focusing on driving or noticing a virtual avatar. The driver's emotional state, such as excitement or confusion, can be inferred by analyzing the emotional characteristics of the driver's voice, such as tone and speech rate, through speech recognition.

[0040] In this embodiment, the vehicle status data may include, but is not limited to, data such as vehicle speed, longitudinal acceleration, angular velocity, and steering angle from the Controller Area Network (CAN) bus, which can be used to characterize the vehicle's driving state. The vehicle speed can be the linear velocity value sampled by the vehicle wheel speed sensor, the longitudinal acceleration can be the acceleration component along the X-axis of the vehicle body output by the inertial measurement unit, the angular velocity can be the yaw angular velocity around the Z-axis of the vehicle body collected by the gyroscope, and the steering angle can be the absolute value of the steering angle output by the steering wheel angle sensor.

[0041] Optionally, vehicle driving data can be obtained through the vehicle's CAN bus. This driving data can be vehicle status perception data, including but not limited to: acceleration and angular velocity (indicating the vehicle's current acceleration or deceleration), vehicle speed and steering angle (to determine the vehicle's driving status, such as sharp turns), etc.

[0042] For example, driver monitoring systems (DMS), eye-tracking systems (OMS), and voice emotion analysis are used to collect user emotion and behavioral data. Facial expressions are analyzed to identify emotions such as pleasure, fatigue, and focus. Eye tracking monitors the driver's gaze point and direction of gaze to determine their focus. Voice recognition systems capture the driver's tone, speed, and other emotional characteristics to identify their emotions.

[0043] For another example, real-time vehicle driving data is collected via the CAN bus and other vehicle sensors. The vehicle's motion status can be used to determine the driver's potential behavioral responses, influencing the adjustment of the virtual avatar's actions. Vehicle acceleration and road surface information help dynamically adjust the virtual avatar's movements (e.g., smoother movements during deceleration, and gentler movements on bumpy roads).

[0044] Through the above step S102, by utilizing multimodal perception data (such as DMS facial emotion, voice emotion, OMS gaze direction) and vehicle CAN bus driving data (such as vehicle speed, acceleration, steering angle, etc.), synchronous perception of the driver's emotional intentions and the physical environment is achieved. This avoids the static response mode that relies solely on command triggers, and provides a real, dynamic, and context-sensitive input basis for subsequent action decisions, thereby improving the scene adaptability and emotional fit of the interaction.

[0045] Step S104: The state perception data and driving data are fused to obtain fused data.

[0046] In the technical solution provided by step S104 of this application, the fused data can be a state tensor, which may include information such as user emotions, vehicle status, and interaction commands.

[0047] Optionally, state perception data from multi-source heterogeneous sensors (such as DMS facial expressions, voice emotion, and OMS eye-tracking information) and vehicle dynamics driving data (such as vehicle speed, acceleration, steering angle, and road bump index) are semantically and temporally aligned and fused to generate a unified, compact, and highly expressive "state tensor," which serves as the input to the Motion Policy Network (MPN). This fusion process is not simply a matter of concatenating the original data; rather, it can be achieved through a lightweight, deployable multimodal feature fusion module on the vehicle's infotainment system.

[0048] Optionally, since different sensors have different sampling frequencies (e.g., 30Hz for DMS and 100Hz for CAN bus), the fusion module can align the source data to a unified time base through timestamp interpolation and a sliding window mechanism. This ensures that emotional states and vehicle dynamic changes are accurately matched on the timeline, avoiding misjudgments due to delays or misalignments. Furthermore, feature extraction and dimensionality reduction can be performed on the state-aware data and driving data. A lightweight attention mechanism (such as sigmoid weighting or a simplified version of multi-head attention in Transformer) can be used to dynamically evaluate the contribution weight of each modality in the current context. A context-aware fusion state tensor is generated through weighted summation.

[0049] Optionally, the fusion module can have a built-in outlier filtering mechanism (such as removing low-confidence facial expression recognition results) and combine it with historical states to perform a moving average, thereby mitigating sensor momentary errors (such as false recognition caused by strong light) and improving the system's stability in complex vehicle cabin environments.

[0050] Optionally, driver emotion perception data (such as facial expressions, voice tone, and eye movement direction) and vehicle driving data (such as vehicle speed, acceleration, and steering angle) are intelligently fused together. Through lightweight feature extraction and weighted integration, a unified state tensor is generated. This state tensor can be used to represent the real-time state of the user's emotions and the driving environment, providing accurate, efficient, and low-redundancy input for the subsequent action strategy network. This ensures that the virtual avatar's actions can synchronously respond to both the human and vehicle contexts, improving the naturalness and adaptability of the interaction.

[0051] Through the above step S104, heterogeneous perception data (i.e., state perception data and driving data) are uniformly encoded into a structured "state tensor" to achieve cross-modal semantic alignment and feature complementarity, effectively eliminate noise interference from a single sensor (such as the effect of lighting on facial expression recognition), and enhance system robustness; at the same time, the computational overhead is reduced through a lightweight fusion module (such as attention weighting), ensuring efficient data aggregation with low latency and low power consumption on the vehicle terminal side, supporting real-time interaction requirements.

[0052] Step S106: Based on the fused data, determine the control instructions for the in-vehicle virtual object to be displayed in the vehicle, wherein the control instructions are used to characterize the display rules of the skeletal movements of the in-vehicle virtual object.

[0053] In the technical solution provided in step S106 of this application, the aforementioned in-vehicle virtual object can be an in-vehicle virtual image displayed in the vehicle, or a digital virtual image presented in the intelligent car cockpit, such as a virtual assistant, an artificial intelligence (AI) partner, or other visual interactive subject used for emotional and anthropomorphic human-computer interaction with the driver or passengers. It can also be a virtual character rendered through graphics and capable of being driven by actions. The aforementioned control instructions can be used to characterize the display rules of the skeletal movements of the in-vehicle virtual object, or can be high-level, parameterized action control signals generated by the action policy network based on the fusion state tensor. These instructions can be used to guide the in-vehicle virtual image to perform actions, and may include, but are not limited to, parameters such as action sequences, speed adjustment coefficients, and fusion weights. The aforementioned action sequence can be a set of predefined, lightweight basic action fragment identifiers (e.g., smiling, nodding, waving), which can be used to represent the actions that the virtual image should perform in sequence. Each action is an independently stored short animation unit, eliminating the need for a complete sequence and significantly saving storage space. The speed adjustment coefficient mentioned above can be a floating-point number between 0.5 and 1.5, which can be used to dynamically adjust the playback speed of action segments and to achieve adaptive synchronization between the rhythm of the action and the user's emotions and vehicle dynamics. The fusion weight mentioned above can be a weighting coefficient, which can be used to control the superposition ratio and transition smoothness between multiple parallel action segments.

[0054] Optionally, using an action policy network, a set of high-level, parameterized control commands can be output based on the fused data generated in step S104, i.e., the state tensor. These control commands can then precisely guide the movement of the in-vehicle virtual avatar. Instead of directly generating image or skeletal animation data, these control commands describe how the virtual avatar should "execute" actions in a lightweight, structured command format, thereby achieving highly natural interaction with low resource consumption.

[0055] Through step S106 above, based on the fused data, the motion strategy network outputs high-level control instructions (motion sequence, speed adjustment coefficient, fusion weight), instead of directly outputting skeletal frame data, thus decoupling the "decision layer" and the "execution layer". This method significantly reduces the instruction size and enables highly flexible, non-repetitive, and context-adaptive motion strategy generation with limited computing power, fundamentally solving the problems of rigid preset animations and resource redundancy.

[0056] Step S108: Obtain multiple skeletal animations that match the control commands.

[0057] In the technical solution provided by step S108 of this application, the skeletal animation can be used to drive the vehicle virtual object to perform dynamic movement. It can be used to represent a hierarchical skeleton structure composed of multiple joints (bones) established for the vehicle virtual object. Then, by defining the position, rotation and scaling transformation of each bone on the time axis (i.e. keyframe animation), the deformation and movement of the entire model can be driven.

[0058] Optionally, skeletal animation indirectly controls the shape of the vehicle virtual object by controlling the skeleton of the vehicle virtual object, which has the advantages of high storage efficiency, strong motion reusability, and easy interpolation and smooth transition.

[0059] For example, based on the control commands output in step S106, skeletal animation segments corresponding to the action identifiers in the control commands are retrieved from the locally pre-built basic skeletal animation library. This basic skeletal animation library can consist of several small-granularity, reusable, standardized skeletal animations. Each animation segment contains only one or a few basic actions, such as "smiling," "nodding slightly," "raising a hand," or "looking away." Each animation is stored as skeletal keyframe data (such as joint rotation matrices, timestamps, interpolation curves, etc.) and does not contain texture, lighting, or complex rendering information to minimize storage overhead. The corresponding animation identifiers are extracted based on the "action sequence" in the control commands, and the corresponding skeletal animation files are retrieved in parallel from the animation library. For example, if the control commands contain "smiling" and "nodding slightly," then two skeletal animation segments correspond to them respectively. This process does not require loading the complete interactive animation flow; it only extracts the necessary basic units as needed, significantly reducing memory usage and loading latency, and meeting the dual constraints of real-time performance and resource constraints on the vehicle-mounted device. In addition, animation data of high-weight actions can be loaded first based on "fusion weight", the sampling precision of weighted actions can be reduced or loading can be delayed, and resource scheduling efficiency can be further optimized.

[0060] The entire process S108 is completed under low power consumption conditions on the device side. It does not require cloud computing or real-time rendering of complex geometry. It only relies on a lightweight animation library and efficient interpolation algorithm to achieve highly free and natural virtual character interaction.

[0061] Through the above step S108, a small number of standardized skeletal animation clips (such as "smiling" and "nodding") are called from the lightweight basic motion library by index, without the need to store a massive number of complete animation sequences, which significantly reduces the storage occupation of the vehicle system (saving more than 70% of animation space); at the same time, the dynamic combination and reuse of actions are realized through the instruction matching mechanism, which greatly improves the reuse efficiency of animation resources and the scalability of the system.

[0062] Step S110: Control the skeletal movements of the vehicle-mounted virtual object according to multiple skeletal animations and control instructions.

[0063] In the technical solution provided by step S110 of this application, multiple skeletal animations obtained through step S108 can be combined with the speed adjustment coefficient and / or fusion weight in the control command output by step S106, and dynamically synthesized and driven in real time through an animation mixing engine to finally generate a natural and coherent virtual image skeletal movement that conforms to the interactive context, thereby achieving the purpose of quickly controlling the movement of the vehicle virtual object according to actual needs.

[0064] In related technologies, large amounts of animation data are typically transmitted and rendered within the vehicle system, consuming significant memory and bandwidth, thus impacting overall performance. In this embodiment, the MPN outputs high-level control commands (which may include motion sequences, speed control coefficients, fusion weights, etc.), rather than complete animation data (such as skeletal data, image frames, etc.). The amount of data transmitted for control commands is very small, reducing the system's bus bandwidth burden and effectively lowering the data transmission load. This ensures the smooth operation of other functions within the vehicle system, improves overall computational efficiency, and achieves the goal of reducing bandwidth and computational burden.

[0065] Through step S110 above, based on the speed adjustment coefficient and fusion weight in the control command, multiple basic animations are interpolated in real time and fused with intensity weighting to achieve smooth transitions and rhythmic adaptation between actions. For example, a slow smile and slight nod when fatigued avoid the "stuttering" and "abrupt jumps" caused by traditional state machine switching. Combined with physical inertia correction, the virtual character's movements are synchronized with the vehicle's movement (such as leaning forward during sudden braking), greatly enhancing the naturalness, immersion, and realism of the interaction.

[0066] Through steps S102 to S110, driver state perception data and vehicle driving data are acquired. The state perception data represents the driver's emotions, and the driving data represents the vehicle's driving state. The state perception data and driving data are fused to obtain fused data. Based on the fused data, control instructions for the in-vehicle virtual object to be displayed are determined. These control instructions represent the display rules for the skeletal movements of the in-vehicle virtual object. Multiple skeletal animations matching the control instructions are acquired. The skeletal movements of the in-vehicle virtual object are controlled according to the multiple skeletal animations and the control instructions. In other words, in this application, control instructions for the in-vehicle virtual object are determined based on the driver's state perception data and the vehicle's driving data. These control instructions can be used to determine the display rules for the skeletal movements of the in-vehicle virtual object, such as display time and display speed. Furthermore, the skeletal animations matching the control instructions can be invoked, and the skeletal movements of the in-vehicle virtual object can be adjusted according to the skeletal animations and the control instructions, thereby achieving the goal of accurately displaying the virtual object. This improves the technical efficiency of virtual object generation and solves the technical problem of low virtual object generation efficiency.

[0067] The above-mentioned method of this application will be further described below.

[0068] As an optional implementation, the method may further include: acquiring multiple interaction commands of the vehicle at multiple consecutive moments, wherein the multiple interaction commands are used to characterize the changing trend of the driver's driving intention; step S104, fusing the state perception data and driving data to obtain fused data, including: fusing the state perception data, driving data, and interaction commands to obtain fused data.

[0069] In this embodiment, the aforementioned interactive commands can be commands obtained by the driver in real time through a voice recognition system, in-vehicle infotainment system, and human-machine interaction logs. These commands may include, but are not limited to, dynamic contextual information such as the content and semantics of the user's voice commands, the currently playing media, navigation status, and active application interfaces. Semantic parsing and intent inference are then performed in conjunction with historical interaction records. Multiple interactive commands at multiple consecutive moments can be interactive context data, which may include, but are not limited to, the content of the voice commands, the results of user gesture recognition, and the beat information of the currently playing music in the vehicle. The aforementioned voice command content is converted into a semantic label sequence through a voice recognition model. The aforementioned gesture recognition results are obtained through a depth camera and a skeletal keypoint detection algorithm. The aforementioned beat information can be beats per minute (BPM), which can be determined by the periodic energy peak frequency extracted by the audio spectrum analysis module.

[0070] Optionally, multiple interaction commands from the vehicle at multiple consecutive moments can be acquired to obtain interaction context data. This interaction context data can be used to determine the dynamic changes in voice commands, vehicle information, and user intent. The content of the user's voice commands and the driver's emotions can be analyzed through a voice recognition system; the driver interacts with the virtual assistant through gestures (such as raising a hand or nodding), therefore, user actions and gestures can be recognized.

[0071] Optionally, contextual information such as the current voice command, music BPM, and vehicle status can be analyzed to assist in generating action decisions. For example, when a driver issues a voice command, the content and tone of the command need to be considered to adjust the virtual avatar's actions accordingly.

[0072] Optionally, multiple interaction commands are acquired to obtain interaction context data. The acquired state perception data, multiple interaction commands, and driving data are then fused to obtain fused data. The collected multiple perception data are merged into a unified "state tensor" through a data fusion unit for subsequent decision processing. The state tensor includes information such as user emotions, vehicle status, and interaction commands.

[0073] As an optional implementation, the state perception data, driving data, and interaction commands are fused to obtain fused data, including: aligning the state perception data, driving data, and interaction commands with timestamps; encoding the time-stamp-aligned state perception data, driving data, and interaction commands respectively to obtain first encoded data corresponding to the state perception data, second encoded data corresponding to the driving data, and third encoded data corresponding to the interaction commands; and fusing the first encoded data, second encoded data, and third encoded data to obtain fused data.

[0074] In this embodiment, the data fusion process can align, denoise, semantically encode, and integrate the state perception data, driving data, and interaction commands from the multi-source heterogeneous perception system, providing high-quality, structured, and temporally consistent input for the subsequent action policy network.

[0075] For example, user state perception data is collected through DMS (Driver Monitoring System), OMS (Eye Tracking System), and microphone arrays. This state perception data may include, but is not limited to, facial expressions, eye fixation points, pupil diameter, blink frequency, and voice emotion characteristics. Vehicle state data is also acquired in real-time via the CAN bus. This vehicle state data can be driving data, including but not limited to, vehicle speed, longitudinal / lateral acceleration, angular velocity, steering angle, suspension vibration amplitude, and road bump index. Simultaneously, multiple interactive commands are acquired, including but not limited to: the current voice command content, the current state of the in-vehicle system, and environmental background information. The current voice command content can be the parsing result of Natural Language Processing (NLP), such as "turn on the air conditioning," and the semantic intent of the voice command content, such as the command type: adjustment, query, entertainment, etc. The current state of the in-vehicle system is determined (e.g., navigation in progress, music playback in progress, autonomous driving mode); and environmental background information such as music BPM (beats per minute) is acquired.

[0076] Furthermore, the acquired data streams can be aligned using timestamps (e.g., based on the system clock or CAN frame synchronization signal) to unify them to the same time base, ensuring consistency of multimodal data in the time dimension and avoiding decision misalignment due to sampling delays. The aligned data can be cleaned and normalized to remove outliers, such as misidentified "smiles" or CAN signal jitter; data of different dimensions can also be normalized. Finally, feature extraction and semantic encoding are performed on the normalized data. A lightweight feature extraction module can be used to perform semantic abstraction on the normalized data, transforming low-level raw signals into high-order semantic features, improving the generalization ability and decision robustness of the MPN, thus obtaining the first, second, and third encoded data. A feature-level fusion method can be used to concatenate or weighted aggregate the semantic feature vectors of different modalities—that is, the first, second, and third encoded data—to form a unified state tensor.

[0077] For example, emotion vectors, speech vectors, vehicle state vectors, and intent vectors can be directly concatenated into a single long vector to obtain fused data. Attention-weighted fusion can also be introduced, employing a lightweight attention mechanism to dynamically allocate weights across modalities. For instance, when a vehicle makes a sharp turn, the weight of vehicle state increases, while the weight of emotion data decreases appropriately; when a user's voice expresses strong emotion, the weight of voice emotion dominates, with other modalities serving as auxiliary. Through these methods, first-coded data, second-coded data, and third-coded data can be fused to obtain fused data.

[0078] Optionally, since virtual avatar interactions have temporal dependencies—for example, users speak continuously and their emotions change gradually—the fusion module can perform sliding window aggregation on the state tensors of the latest N frames (e.g., N=5) to form a temporal state tensor. Before inputting into the MPN, long-range dependencies can be further modeled using an encoder (e.g., LSTM or Transformer) to capture emotional evolution trends such as "users moving from calm to irritability." The final output is a structured, multidimensional, and temporally aware low-dimensional state tensor.

[0079] In this embodiment, state perception data, driving data, and multiple interaction commands are multimodally complementary to avoid misjudgments from a single sensor (such as misjudging "excitement" based solely on voice when it is actually noise interference). After fusion, the data is compressed into a low-dimensional tensor, which is much smaller than the original image or audio data, saving memory and bandwidth.

[0080] Optionally, the generation of fused data includes time alignment and normalization of multimodal data, with a sampling frequency of not less than 20 Hz to ensure the real-time and synchronous nature of motion generation. The time alignment uses the Dynamic Time Warping (DTW) algorithm to align the sampling timestamps of the camera, microphone, CAN bus, and gesture recognition module. The normalization process uses Z-score standardization to map each modal data to a distribution space with a mean of 0 and a variance of 1.

[0081] As an optional implementation, step S108, acquiring multiple skeletal animations that match the control instructions, includes: determining multiple skeletal animations that match the action sequence from the basic action library according to the action sequence in the control instructions, wherein the action sequence includes multiple sub-actions, and the sub-actions correspond to the skeletal animations; and acquiring multiple skeletal animations sequentially according to the storage order of the multiple sub-actions.

[0082] In this embodiment, the aforementioned basic motion library can be a library storing basic motion fragments, which can be small-granular motion units (such as nodding, opening eyes, raising hands, etc.). Each basic motion fragment is pre-designed and rendered, and stored in the basic motion library in digital form.

[0083] Optionally, after determining the control instructions, the motion control engine can be invoked to select basic motion segments and execute them sequentially based on the MPN output, i.e., the control instructions. For example, if the MPN outputs the motion sequence as "smile, nod slightly", the motion control engine can invoke these two motion segments (i.e., skeletal animation) in sequence.

[0084] Optionally, the control instructions may include a sequence of actions to be retrieved, which can be an action sequence identifier used to determine the skeletal animation to be retrieved. This sequence may include indices of predefined basic motion segments and may include multiple sub-actions. The skeletal animation to be retrieved can be determined according to the sub-actions, and the retrieval order of the skeletal animation can be determined based on the order of the multiple sub-actions.

[0085] Optionally, the aforementioned basic movement segments may include smiling, nodding, waving, slightly shaking the head, eye contact, and slight head tilting. Among them, smiling corresponds to the combined movement sequence of chin elevation and zygomatic muscle contraction; nodding corresponds to the periodic movement of cervical spine flexion and extension; waving corresponds to the coordinated movement of shoulder abduction and elbow flexion and extension; eye contact corresponds to the synchronous movement of pupil orientation adjustment and head fine-tuning in the eyeball model; and slight head tilting corresponds to the combined movement of cervical spine lateral tilt and asymmetrical shoulder posture.

[0086] Optionally, each basic motion fragment in the basic motion library is pre-rendered skeletal animation data, containing a joint rotation matrix sequence, and each fragment is less than 1.5 seconds long, stored in a binary compressed format; the joint rotation matrix sequence is based on a local coordinate system, with a sampling frequency of 60Hz per frame, and the compression format can adopt a hybrid format based on quaternion differential coding and entropy coding.

[0087] Optionally, based on the action sequence, a basic action library can be determined, containing multiple skeletal animations that match the action sequence. The corresponding skeletal animations can be retrieved according to the stored order of the action sequence to control the vehicle-mounted virtual object to move according to the skeletal animation.

[0088] In this embodiment, by introducing MPN to control the in-vehicle virtual avatar, the problems of high resource consumption and stiff action switching in traditional virtual avatar interaction systems are solved, effectively improving the flexibility and interactive experience of the virtual avatar.

[0089] Alternatively, from the perspective of resource consumption and system performance optimization, traditional in-vehicle virtual avatar systems rely on a large number of preset animation files (such as sequence frames, skeletal animation, etc.) to realize various actions of the virtual avatar. These files require a large amount of storage space, and the storage requirements become even larger when multiple interactive scenarios need to be covered. In this embodiment, by adopting a hybrid mode of basic animation library and AI-driven motion recombination and speed adjustment, only a small basic motion fragment library needs to be stored, while the richness and variability of the motion are dynamically generated by MPN. This greatly reduces the storage requirements of animation resources and significantly saves the storage space of the vehicle system, thereby achieving the goal of reducing storage space requirements.

[0090] As an optional implementation, the skeletal movements of the vehicle-mounted virtual object are controlled according to multiple skeletal animations and control instructions, including: scaling the skeletal animations on the time axis according to the speed adjustment coefficients corresponding to the skeletal animations in the control instructions, wherein the speed adjustment coefficients are used to adjust the playback speed of the skeletal animations; performing smooth blending processing on the scaled multiple skeletal animations according to the fusion weight parameters in the control instructions to obtain a composite skeletal animation trajectory, wherein the fusion weight parameters are used to characterize the superposition ratio of the multiple skeletal animations; and controlling the skeletal movements according to the composite skeletal animation trajectory.

[0091] In this embodiment, the aforementioned speed adjustment coefficient can be used to adjust the playback speed of the skeletal animation. For example, the speed of the nodding motion can be slowed down to 70% of the normal speed. The aforementioned fusion weight parameter, which can be simply referred to as fusion weight, can be used to characterize the superposition ratio of multiple skeletal animations. For the transition between multiple action segments, the motion control engine can smoothly fuse multiple action segments according to the fusion weight to ensure smooth motion.

[0092] Optionally, this embodiment can also generate corresponding skeletal animations based on input control commands, and adjust the posture of the virtual character through inverse kinematics to make the skeletal animations conform to realistic physical representations. For example, during sudden braking, the character's movement will show a tilt, enhancing realism.

[0093] Optionally, the speed adjustment coefficient can be a floating-point number between 0.5 and 1.5, which can be used to linearly scale the playback duration of the basic action segment. This coefficient can be dynamically generated based on the weighted fusion value of the emotion tag and the vehicle acceleration. The emotion tag is determined by the maximum likelihood estimation of the discrete probability distribution output by the multimodal emotion classifier, the vehicle acceleration is the absolute value of the longitudinal acceleration, and the weighted fusion value is the sum of the emotion tag confidence multiplied by the first weight coefficient and the vehicle acceleration multiplied by the second weight coefficient.

[0094] Optionally, the fusion weight is the superposition ratio of multiple action segments on the time axis, and the calculation basis may include emotion intensity, action semantic relevance and environmental vibration level; emotion intensity is represented by the inverse of the entropy value output by the emotion classifier, action semantic relevance is calculated by the shortest path distance between nodes in the predefined action semantic map, and environmental vibration level is determined by the proportion of the main frequency energy extracted after the vibration amplitude collected by the suspension sensor is filtered in the frequency domain.

[0095] Optionally, the dynamic superposition process of the fusion weights can employ a sigmoid smoothing interpolation function to achieve a smooth transition during the intersection of adjacent action segments, avoiding abrupt changes in action. The sigmoid interpolation function can be a logistic function, with the input being the normalized time ratio within the intersection period and the output being the superposition ratio. The inflection point of the function is located at the midpoint of the intersection period, and the slope is controlled by the absolute value of the gradient of the fusion weights.

[0096] In this embodiment, to enhance the flexibility and realism of interactive actions and increase their non-repetitiveness and flexibility, MPN generates real-time action sequences based on environmental, emotional, and voice data, and controls the rhythm and transitions of the actions through speed adjustment coefficients and fusion weights. The virtual avatar's actions are no longer fixed and repetitive, but change according to the specific context of each interaction, demonstrating high flexibility and personalization, and enhancing the naturalness and emotional expression of the interaction.

[0097] Optionally, in traditional interactive systems, the transitions between actions of virtual avatars can be abrupt, such as "stuttering" or "jumping" from one action to another. By integrating weights and dynamically adjusting speed, MPN can control the transitions between multiple action segments in real time, making in-vehicle virtual objects smoother and more natural. The transition time between different action segments (such as nodding and smiling) can be dynamically calculated by MPN to ensure seamless connections between actions, enhancing the realism and immersion of the interaction, thereby achieving the goal of increasing natural action fusion and transitions.

[0098] Optionally, traditional in-vehicle virtual avatars neglect the impact of the vehicle's physical environment on user behavior. For example, the avatar's movements cannot adaptively adjust during sudden braking or vehicle vibration. In this embodiment, the MPN can combine data such as vehicle acceleration and suspension status to generate movements that adapt to environmental changes. For instance, when the vehicle brakes suddenly or travels on a bumpy road, the virtual avatar's movements will take physical inertia and vibration into account, adjusting the speed, force, and direction of the movements to better match the real environment and enhance immersion.

[0099] For example, the motion execution engine retrieves corresponding basic animation clips from the locally stored basic animation library based on the "motion sequence" (e.g., smiling, slight nodding) output by the MPN. Each basic animation clip can be independent, standardized skeletal animation data, which may include keyframe data of the virtual character's joints (e.g., head, shoulders, arms) rotation and displacement on the timeline. The motion execution engine scales the timeline of each skeletal animation according to the speed adjustment factor (e.g., 0.7) output by the MPN. If the speed adjustment factor is 0.7, the original 1-second "smile" animation is extended to approximately 1.43 seconds for a smoother rhythm; if it is 1.2, it is compressed to 0.83 seconds for a faster pace. This step can be achieved through time remapping technology, without changing the animation data itself, only adjusting the playback rate, ensuring resource reuse and efficiency.

[0100] Furthermore, for the fusion weights of the MPN output (e.g., 70% for a smile + 30% for a nod), weighted animation blending can be performed. This aligns the two animation clips, "smile" and "slight nod," with their starting frames in time, and then superimposes skeletal transformation data (rotation quaternions, translation vectors) according to their weights. During the animation transition phase (e.g., the first 0.3 seconds), linear interpolation or spherical linear interpolation can be used to smoothly blend the joint poses of the two animations, avoiding jitter or stuttering caused by "hard switches." The resulting blend is a composite skeletal animation trajectory, which can represent a natural, coordinated movement "primarily a smile, secondarily a nod." Skeletal movements can be controlled according to this composite skeletal animation trajectory.

[0101] In this embodiment, to enhance the realism and environmental adaptability of the virtual character's movements, the engine can introduce an inverse kinematics module for fine-tuning the posture after mixing the basic animation. Simultaneously, a physical consistency check can be introduced to verify whether joint angles exceed reasonable biomechanical ranges (e.g., the maximum neck rotation angle does not exceed 45°), preventing unnatural postures such as "reverse joints."

[0102] Optionally, the above process does not regenerate the original animation. Instead, it uses a lightweight basic animation library to achieve highly natural, environmentally adaptive, and low-computing-power-constrained virtual character movement performance through parametric control and intelligent correction, perfectly suited for resource-constrained scenarios on the vehicle side.

[0103] As an optional implementation, the method may further include: performing inertial correction on the skeletal motion according to driving data; and adjusting the skeletal motion according to the changed state perception data in response to a change in state perception data.

[0104] In this embodiment, the skeletal motion can also be inertially corrected based on driving data. By physically correcting the skeletal motion, the realism of the in-vehicle virtual object's movement can be improved. Furthermore, if the state perception data changes, the skeletal motion can be adjusted based on the updated state perception data.

[0105] Optionally, the animation rendering engine performs final graphics rendering on the posture information output by the motion control module, utilizing the vehicle's graphics processing unit (GPU) to efficiently generate the visual effects of the virtual avatar. Simultaneously, it can perform physical corrections on the skeletal movements of the in-vehicle virtual objects. The physical correction process can include inertial compensation and geometric correction.

[0106] Optionally, a physics engine can be used to determine the vehicle's acceleration and motion state based on driving data. Using this acceleration and motion state, inertial corrections can be made to the virtual avatar's movements. For example, during sudden braking, the virtual avatar's body will lean forward, exhibiting a realistic reaction. Inverse kinematics can then be used to geometrically correct the virtual avatar to ensure its posture aligns with the target, such as ensuring that the head, eyes, and gestures accurately point in the user's line of sight or gesture direction.

[0107] Optionally, the virtual avatar's actions can be adjusted based on real-time interactive feedback. For example, if the driver changes their instructions or mood, the virtual avatar's actions can be adjusted instantly, generating a new action sequence, and the speed adjustment coefficient and fusion weights can be readjusted. Furthermore, the MPN weights can be updated based on user feedback and new interactive needs through an over-the-air (OTA) mechanism to continuously improve the virtual avatar's performance. New action styles can be implemented by updating the model without replacing a large number of animation files.

[0108] In this embodiment, the motion execution engine incorporates an inverse kinematics (IK) algorithm to physically correct the skeletal posture of the virtual avatar based on the vehicle's acceleration and inertial direction, ensuring that the motion response conforms to realistic inertial effects. The IK algorithm uses the pelvic center as a reference base and solves for joint angular displacements using gradient descent, aligning the virtual force vector at the foot-seat contact point with the vehicle's acceleration direction. The correction is linearly mapped from the projection magnitude of the acceleration vector in the vehicle's coordinate system to the joint rotation offset. Furthermore, the model weights of the motion policy network can be updated via an OTA (Over-The-Air) mechanism to adapt to new interaction modes without updating the animation files in the basic motion library. This OTA update can be implemented via differential packet transmission, transmitting only the incremental parameters of the model weights. The update process is performed while the vehicle is stationary, and the model integrity is ensured after the update through hash verification and signature verification.

[0109] Optionally, when a user command interruption or a new voice trigger is detected, the action policy network can reset the action sequence in real time and dynamically adjust the speed adjustment coefficient and fusion weight to achieve seamless continuation of action playback. Interruption detection is triggered by the duration of the silent segment output by the Voice Activity Detection (VAD) module exceeding the threshold, and new voice trigger is triggered by the confidence level output by the keyword recognition model exceeding the threshold. The reset process maintains the skeletal state of the current action segment as the initial state, and the new action sequence is interpolated and connected from the current timestamp.

[0110] As an optional implementation, step S104, based on the fused data, determines the control command for the in-vehicle virtual object to be displayed in the vehicle, including: mapping the fused data to obtain an action sequence; identifying the fused data to determine the driver's emotion and the vehicle's driving state; using the emotion and driving state to determine a speed adjustment coefficient matching the action sequence, and using the emotion to determine a fusion weight parameter matching the action sequence; combining the action sequence, the speed adjustment coefficient matching the action sequence, and the fusion weight parameter matching the action sequence to obtain the control command.

[0111] In this embodiment, the fused data can be input into an action policy network, which processes the fused data to obtain corresponding control commands. The action policy network can be a lightweight deep learning model, employing a Long Short-Term Memory Network (LSTM) or Transformer architecture. The input is the fused state tensor, and the output is a three-dimensional control parameter vector, which can be deployed on the vehicle's infotainment system side. The LSTM architecture can contain two layers of bidirectional LSTM units, with no more than 128 hidden units per layer. The Transformer architecture can employ a single-head attention mechanism with a 64-dimensional attention head and a 256-dimensional feedforward hidden layer. Model parameters are obtained from a teacher model through quantization compression and knowledge distillation techniques.

[0112] Optionally, the action policy network is obtained through offline training. The training data includes labeled multimodal interaction scenarios and corresponding manually designed action parameters. The training objective function minimizes the dynamic time warping distance between generated actions and expert actions. Each sample in the training data contains a time-aligned multimodal input sequence and a corresponding manually labeled action segment sequence. The DTW distance calculation uses Euclidean distance as the local cost function, and the path constraint uses a Sakoe-Chiba window with a width of 15% of the input sequence length.

[0113] Optionally, the action sequence consists of multiple basic action segments (such as nodding, waving, etc.). The MPN generates an action sequence suitable for the current interaction scenario based on the input multimodal data (i.e., fused data). Specifically, the MPN receives a state tensor as input and analyzes the input fused data using a pre-trained neural network model. It can also predict appropriate virtual avatar actions based on the driver's emotion tags and voice sentiment analysis results. For example, if the driver appears tired, a relaxing and soothing action sequence (such as a slight nod or smile) can be selected.

[0114] Optionally, based on the current interaction context, a matching basic action fragment can be selected, such as: a happy emotion corresponds to smiling, nodding, and waving; a tired emotion corresponds to slight nodding, bowing, and slowly shaking the head; and a tense emotion corresponds to fixed eye contact and slightly tilting the head.

[0115] Optionally, the action sequence output by MPN can contain a series of identifiers or indices for basic action segments for subsequent animation generation. For example, assuming the system analyzes and determines that the user's emotion is "fatigue," and combines this with vehicle speed and road conditions (such as vibration), MPN might select the following action sequence: [smile, nod]: a slow nodding motion combined with a smile.

[0116] Optionally, a speed adjustment coefficient needs to be generated. This coefficient controls the playback speed of the basic action segments to adjust the rhythm of the movements. MPN generates the speed adjustment coefficient by analyzing factors such as driver emotions and vehicle driving conditions. The required rhythm of the movements can be predicted based on the driver's emotions (e.g., fatigue, excitement). For example, when fatigued, the movement rhythm should be slowed down, and the speed adjustment coefficient should be lower. Furthermore, factors such as vehicle acceleration, deceleration, and vibration also affect the speed adjustment coefficient. For instance, on bumpy roads, the speed adjustment coefficient may be lowered to make the movements appear smoother, adapting to environmental changes. If the driver's emotions are strong (e.g., eagerness, joy), the speed adjustment coefficient will be higher, and the movements will be faster and more powerful.

[0117] Optionally, a speed adjustment coefficient can be output based on emotions and driving status. This coefficient can be a floating value between 0.5 and 1.5, which can be used to determine the playback speed of the actions. For example, in a fatigued state, the speed adjustment coefficient = 0.7 (indicating slower actions and milder emotions), and in an excited state, the speed adjustment coefficient = 1.2 (indicating faster actions and stronger emotions). The generation of the speed adjustment coefficient ensures that the rhythm of the virtual avatar's actions matches the driver's emotional state and driving environment.

[0118] Optionally, the fusion weight is used to control the superposition ratio and transition duration of multiple basic action segments on the timeline, ensuring natural transitions between actions and avoiding abrupt switches. Therefore, the weight of each action segment can be calculated based on the driver's emotions and interaction intentions. For example, in a state of fatigue, a smiling action may have a higher weight, a nodding action a lower weight, and the smile may be maintained for a longer period. Considering the interrelationships between different actions, such as the fusion of nodding and smiling, a smooth transition between the two needs to be ensured. MPN outputs a fusion weight parameter, which determines the proportion and transition between multiple actions. For example, in a state of fatigue, the fusion weight = [nodding: 30%, smiling: 70%], with the smiling action having a higher weight and a longer duration; in a state of happiness, the fusion weight = [nodding: 50%, waving: 50%], with the proportions of the two actions being equal.

[0119] Optionally, the goal of weight fusion generation is to achieve smooth action transitions and natural combinations of multiple actions, ensuring that the virtual avatar (i.e., virtual object) remains fluid and unobtrusive when performing multiple actions. By comprehensively analyzing the state tensor, parameters such as action sequences, speed adjustment coefficients, and fusion weights are generated to drive the virtual avatar to perform precise actions. The core of the entire process lies in data fusion and model training, and by adjusting these control parameters in real time, it ensures that the virtual avatar displays natural and emotionally resonant actions in different interactive scenarios.

[0120] Optionally, the fused state tensor is passed as input to the MPN. The MPN model is a deep learning model that can employ an LSTM or Transformer architecture to process temporal data and contextual information. Based on the state tensor, the MPN generates an action sequence suitable for the current interaction. This action sequence can consist of basic action fragments (such as nodding, waving, smiling, etc.), for example, selecting "slight nod" and "smile" when experiencing fatigue. Simultaneously, the MPN can output a speed adjustment coefficient based on the user's mood and vehicle status (e.g., acceleration, bumps, etc.). This speed adjustment coefficient determines the playback speed of the action, adjusting the rhythm of the virtual avatar's movements. For example, a speed adjustment coefficient of 0.6 indicates a slower movement when experiencing fatigue, while a speed adjustment coefficient of 1.2 indicates a faster and more energetic movement when experiencing a happy mood. Finally, the MPN can output control commands including the action sequence, speed adjustment coefficient, and fusion weights.

[0121] For example, MPN maps the state tensor to a set of discrete sequences of basic action identifiers. For instance, when the state tensor identifies "user fatigue + slow speech + smooth driving," MPN outputs the action sequence: smiling, slightly nodding; when it identifies "user excitement + urgent instructions + accelerating," it outputs the action sequence: waving, looking up. This process does not directly generate animation; instead, it intelligently selects the most matching combination from a pre-set basic action library based on semantic context, achieving context-aware dynamic action decision-making.

[0122] Optionally, MPN can input the fused state tensor (including emotion intensity, speech rate, vehicle speed, longitudinal acceleration, road bump index, etc.) into the network's regression branch, and automatically learn the complex relationship between "emotion-speed" and "environment-rhythm" through the trained nonlinear mapping function.

[0123] The technical solutions of the embodiments of the present invention will be illustrated below with reference to preferred embodiments.

[0124] In this embodiment, a method and system for generating in-vehicle virtual avatar actions based on an AI action policy network are proposed. In particular, a lightweight action policy network deployed on the edge is involved. This method achieves a technical solution for generating natural and low-resource-consumption actions of in-vehicle virtual avatars by making decision inferences from multi-source state information and outputting action control parameters.

[0125] Optionally, this embodiment constructs a hierarchical motion generation mechanism of "perception → strategy → parameterized control", the core of which lies in the motion strategy network. This network generates a high-level motion control strategy for the virtual avatar by processing and analyzing multimodal perception data. The network does not directly generate motion data for each frame, but outputs control commands, including motion sequences, speed adjustment coefficients, fusion weights, etc. These control commands are then transformed into specific actions of the virtual avatar by the motion generation module.

[0126] Optionally, in the interactive system of the in-vehicle virtual avatar, MPN can analyze multimodal perception data and combine it with various data inputs such as voice, facial expressions, gestures, and vehicle status to determine the user's emotions and interaction intentions. At the same time, based on the current emotions and interaction background, corresponding high-level control commands (also known as action control strategies) can be generated to drive the virtual avatar to perform corresponding actions.

[0127] Figure 2 This is a flowchart of a method for generating in-vehicle virtual avatar actions based on an AI action policy network, according to an embodiment of this application. Figure 2 As shown, the method may include the following steps.

[0128] Step S202, Data collection and status coding.

[0129] In this embodiment, user emotion and behavior data can be collected through devices such as DMS and OMS to obtain driver state perception data. Simultaneously, real-time vehicle data can be collected through the CAN bus and other sensors in the vehicle to obtain vehicle driving data.

[0130] In step S204, the action policy network processes the data and generates control commands.

[0131] In this embodiment, the action policy network outputs control commands including action sequences, speed adjustment coefficients, and fusion weights based on the collected fusion data.

[0132] Step S206: Execute control commands.

[0133] In this embodiment, the motion control engine can select basic motion segments and execute them in sequence based on the output of MPN, and can also generate corresponding animations based on the speed adjustment coefficient and fusion weight.

[0134] Step S208, Rendering and Physical Correction.

[0135] In this embodiment, the animation rendering engine is responsible for the final graphics rendering of the posture information output by the motion control module, and performs inertial correction on the virtual image's movements using the vehicle system status information.

[0136] Step S210: Interactive feedback and updates.

[0137] In this embodiment, the virtual avatar's actions can also be adjusted based on real-time interactive feedback, thereby updating the MPN weights based on user feedback and new interactive needs to continuously improve the virtual avatar's performance.

[0138] Figure 3 This is a flowchart of an in-vehicle virtual avatar interaction method based on an action policy network according to an embodiment of this application, such as... Figure 3 As shown, the method may include the following steps.

[0139] Step S302: Collect multimodal sensing data.

[0140] In this embodiment, multi-source perception data is collected in real time through onboard sensors, including: user emotion data, which is obtained by recognizing facial expressions through a driver monitoring system, acquiring gaze direction through an eye-tracking system, and extracting features such as voice tone and speech rate through a voice emotion analysis module; vehicle status data, which is obtained through the CAN bus, including vehicle speed, longitudinal / lateral acceleration, steering angle, and road vibration information; and interaction context data, including voice command content, user gesture recognition results, and environmental context information such as music beats (BPM).

[0141] Step S304: Construct the state tensor.

[0142] In this embodiment, the collected multimodal data can be normalized, time-aligned, and feature-encoded to generate a unified, high-dimensional state tensor, which serves as the input to the subsequent action policy network. This state tensor includes user emotion labels, vehicle dynamic parameters, and semantic vectors of interaction intent.

[0143] Step S306: Use the action policy network to perform data reasoning to obtain control commands.

[0144] In this embodiment, the state tensor is input into a pre-trained action policy network, which is a deep neural network based on LSTM or Transformer architecture, to output three sets of high-level action control parameters.

[0145] Step S308, Action execution and parameterization drive.

[0146] In this embodiment, the motion execution engine retrieves the corresponding basic motion segments from the local basic motion library according to the control parameters output by the MPN, and applies a speed adjustment coefficient to adjust the playback rate of the skeletal animation. At the same time, it performs smooth interpolation and time axis overlap fusion on multiple motions according to the fusion weight to generate a continuous and natural composite motion sequence.

[0147] Step S310, Physical Correction and Real-Time Rendering.

[0148] In this embodiment, the fused motion sequence can be transmitted to the motion control module, and the skeletal posture of the virtual image can be adjusted by inverse kinematics (IK). Dynamic compensation is also performed by combining physical parameters such as vehicle acceleration and inertia (e.g., leaning forward during sudden braking and swaying slightly during bumps) to ensure that the motion conforms to real physical laws. Subsequently, the GPU drives the graphics rendering engine to generate a high-fidelity visual output of the virtual image, which is then displayed on the in-vehicle display screen.

[0149] Step S312: Interactive feedback and model update.

[0150] In this embodiment, real-time user feedback (e.g., new instructions, emotional changes, gaze shifts) is continuously monitored to dynamically trigger the aforementioned closed-loop update. Simultaneously, user interaction data is transmitted back to the cloud via an OTA mechanism to continuously optimize MPN model weights and action library content, thereby enabling system self-evolution.

[0151] In this embodiment, control commands for the in-vehicle virtual object are determined based on the driver's state perception data and the vehicle's driving data. These control commands can be used to determine the display rules for the skeletal movements of the in-vehicle virtual object, such as display time and display speed. Furthermore, a skeletal animation matching the control commands can be invoked, and the skeletal movements of the in-vehicle virtual object are adjusted according to the skeletal animation and control commands, thereby achieving the goal of accurately controlling the display of the virtual object. This improves the technical efficiency of virtual object generation and solves the technical problem of low virtual object generation efficiency.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0153] According to an embodiment of this application, an embodiment of a control system for in-vehicle virtual objects is also provided. It should be noted that this system can be used to run the control method for in-vehicle virtual objects described above.

[0154] According to an embodiment of this application, a control device for a vehicle-mounted virtual object is also provided. It should be noted that this device can be used to run the control method for the vehicle-mounted virtual object described above.

[0155] Figure 4 This is a schematic diagram of a control device for an in-vehicle virtual object according to an embodiment of this application. Figure 4As shown, the control device for the in-vehicle virtual object in the vehicle may include: a first acquisition unit 402, a fusion unit 404, a determination unit 406, a second acquisition unit 408, and a control unit 410.

[0156] The first acquisition unit 402 is used to acquire the driver's state perception data and the vehicle's driving data, wherein the state perception data is used to characterize the driver's emotions and the driving data is used to characterize the vehicle's driving state.

[0157] The fusion unit 404 is used to fuse state perception data and driving data to obtain fused data;

[0158] The determining unit 406 is used to determine the control instructions for the in-vehicle virtual object to be displayed in the vehicle based on the fused data, wherein the control instructions are used to characterize the display rules of the skeletal motion of the in-vehicle virtual object;

[0159] The second acquisition unit 408 is used to acquire multiple skeletal animations that match the control instructions;

[0160] The control unit 410 is used to control the skeletal movements of the on-board virtual object according to multiple skeletal animations and control commands.

[0161] Furthermore, the device can also be used to acquire multiple interactive commands of the vehicle at multiple consecutive moments, wherein the multiple interactive commands are used to characterize the changing trends of the driver's driving intentions.

[0162] Furthermore, the fusion unit 404 may include a fusion module for fusing state perception data, driving data, and interaction commands to obtain fused data.

[0163] Furthermore, the fusion module may include: an alignment submodule, used to align the state perception data, driving data, and interaction commands with timestamps; an encoding submodule, used to encode the state perception data, driving data, and interaction commands after timestamp alignment, respectively, to obtain first encoded data corresponding to the state perception data, second encoded data corresponding to the driving data, and third encoded data corresponding to the interaction commands; and a fusion submodule, used to fuse the first encoded data, second encoded data, and third encoded data to obtain fused data.

[0164] Furthermore, the second acquisition unit 408 may include: a determining module, configured to determine multiple skeletal animations matching the action sequence from a basic action library according to the action sequence in the control instruction, wherein the action sequence includes multiple sub-actions, and the sub-actions correspond to the skeletal animations; and an acquisition module, configured to acquire multiple skeletal animations sequentially according to the storage order of the multiple sub-actions.

[0165] Furthermore, the control unit 410 includes: a scaling module, used to scale the skeletal animation along the time axis according to the speed adjustment coefficient corresponding to the skeletal animation in the control command, wherein the speed adjustment coefficient is used to adjust the playback speed of the skeletal animation; a processing module, used to perform smooth blending processing on the scaled multiple skeletal animations according to the fusion weight parameter in the control command to obtain a composite skeletal animation trajectory, wherein the fusion weight parameter is used to characterize the superposition ratio of the multiple skeletal animations; and a control module, used to control the skeletal movements according to the composite skeletal animation trajectory.

[0166] Furthermore, the device can also be used to make inertial corrections to skeletal movements based on driving data; and to adjust skeletal movements according to the changed state perception data in response to changes in state perception data.

[0167] Furthermore, the determining unit 406 may further include: a mapping module for mapping the fused data to obtain an action sequence; a first determining module for identifying the fused data to determine the driver's emotion and the vehicle's driving state; a second determining module for using the emotion and driving state to determine a speed regulation coefficient matching the action sequence, and using the emotion to determine a fusion weight parameter matching the action sequence; and a combining module for combining the action sequence, the speed regulation coefficient matching the action sequence, and the fusion weight parameter matching the action sequence to obtain a control command.

[0168] In the vehicle-mounted virtual object control device of this embodiment, a first acquisition unit acquires the driver's state perception data and the vehicle's driving data, wherein the state perception data is used to represent the driver's emotions and the driving data is used to represent the vehicle's driving state; a fusion unit fuses the state perception data and the driving data to obtain fused data; a determination unit determines the control instructions for the vehicle-mounted virtual object to be displayed based on the fused data, wherein the control instructions represent the display rules of the skeletal movements of the vehicle-mounted virtual object; a second acquisition unit acquires multiple skeletal animations that match the control instructions; and a control unit controls the skeletal movements of the vehicle-mounted virtual object according to the multiple skeletal animations and the control instructions, thereby achieving the technical effect of improving the virtual object generation efficiency and solving the technical problem of low virtual object generation efficiency.

[0169] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application.

[0170] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0171] Embodiments of this application also provide a computer program product, including a computer program that implements the methods of various embodiments of this application when run by a processor.

[0172] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program, which is executed by a processor to implement the methods in various embodiments of this application.

[0173] Embodiments of this application also provide a computer program that, when run by a processor, implements the methods described in the various embodiments of this application.

[0174] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0175] Optionally, Figure 5 This is a structural block diagram of a computer terminal according to an embodiment of the present invention, such as... Figure 5 As shown, the computer terminal 508 may include one or more (only one is shown in the figure) processors 502, memory 504, and transmission devices 506.

[0176] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the control method and device for in-vehicle virtual objects in the vehicle in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned control method for in-vehicle virtual objects. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to computer terminal 508 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0177] The processor can access the information and application programs stored in the memory via the transmission device to execute the steps in the control method for the on-board virtual object in the vehicle described above.

[0178] Those skilled in the art will understand that Figure 5The structure shown is for illustrative purposes only. The computer terminal 508 can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile internet device (MID), a PAD, or other terminal device. Figure 5 This does not limit the structure of the computer terminal 508 described above. For example, the computer terminal 508 may also include components that are more advanced than those described above. Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0179] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0180] Embodiments of this application may also provide an electronic device, which may include a memory and a processor.

[0181] Figure 6 This is a block diagram of an electronic device for a method of controlling an in-vehicle virtual object in a vehicle according to an embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0182] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory 603 from storage unit 608. RAM 603 can also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0183] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0184] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as data verification methods. For example, in some embodiments, the data verification method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the data verification method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform a data verification method by any other suitable means (e.g., by means of firmware).

[0185] According to an embodiment of this application, a method for controlling in-vehicle virtual objects in a vehicle is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0186] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0187] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0188] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0189] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display, monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or pathball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0190] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0191] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0192] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0193] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0194] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection can be through some interfaces; the indirect coupling or communication connection of units or modules can be electrical or other forms.

[0195] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0197] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0198] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for controlling an in-vehicle virtual object, characterized in that, include: Acquire driver state perception data and vehicle driving data in the vehicle, wherein the state perception data is used to characterize the driver's emotions and the driving data is used to characterize the vehicle's driving state. The state perception data and the driving data are fused together to obtain fused data; Based on the fused data, control instructions for the in-vehicle virtual objects to be displayed in the vehicle are determined, wherein the control instructions are used to characterize the display rules of the skeletal movements of the in-vehicle virtual objects; Obtain multiple skeletal animations that match the control commands; The skeletal movements of the in-vehicle virtual object are controlled according to the multiple skeletal animations and the control commands.

2. The method according to claim 1, characterized in that, The method further includes: The vehicle acquires multiple interaction commands at multiple consecutive moments, wherein the multiple interaction commands are used to characterize the changing trend of the driver's driving intention; The process of fusing the state perception data and the driving data to obtain fused data includes: The state perception data, the driving data, and the interaction commands are fused together to obtain the fused data.

3. The method according to claim 2, characterized in that, The process of fusing the state perception data, the driving data, and the interaction commands to obtain the fused data includes: The state perception data, the driving data, and the interaction commands are timestamped and aligned. The state perception data, the driving data, and the interaction command, after being aligned with timestamps, are encoded to obtain the first encoded data corresponding to the state perception data, the second encoded data corresponding to the driving data, and the third encoded data corresponding to the interaction command. The first encoded data, the second encoded data, and the third encoded data are fused together to obtain the fused data.

4. The method according to claim 1, characterized in that, The acquisition of multiple skeletal animations matching the control command includes: According to the action sequence in the control command, a plurality of skeletal animations matching the action sequence are determined from the basic action library, wherein the action sequence includes a plurality of sub-actions, and the sub-actions correspond to the skeletal animations; According to the storage order of the multiple sub-actions, the multiple skeletal animations are obtained sequentially.

5. The method according to claim 4, characterized in that, The step of controlling the skeletal movements of the in-vehicle virtual object according to the multiple skeletal animations and the control commands includes: According to the speed adjustment coefficient corresponding to the skeletal animation in the control command, the skeletal animation is scaled on the time axis, wherein the speed adjustment coefficient is used to adjust the playback speed of the skeletal animation; According to the fusion weight parameter in the control instruction, the scaled multiple skeletal animations are smoothly blended to obtain a composite skeletal animation trajectory, wherein the fusion weight parameter is used to characterize the superposition ratio of the multiple skeletal animations; Control the bone movements according to the composite bone animation trajectory.

6. The method according to claim 5, characterized in that, The method further includes: Based on the driving data, the skeletal movements are inertially corrected; In response to a change in the state-aware data, the skeletal motion is adjusted according to the changed state-aware data.

7. The method according to claim 1, characterized in that, The step of determining the control commands for the in-vehicle virtual objects to be displayed in the vehicle based on the fused data includes: The fused data is mapped to obtain an action sequence; The fused data is used to identify and determine the driver's emotions and the vehicle's driving status; Using the emotion and the driving state, a speed adjustment coefficient matching the action sequence is determined, and using the emotion, a fusion weight parameter matching the action sequence is determined. The control command is obtained by combining the action sequence, the speed regulation coefficient matching the action sequence, and the fusion weight parameter matching the action sequence.

8. A control device for an in-vehicle virtual object, characterized in that, include: The first acquisition unit is used to acquire the driver's state perception data and the vehicle's driving data, wherein the state perception data is used to characterize the driver's emotions and the driving data is used to characterize the vehicle's driving state. The fusion unit is used to fuse the state perception data and the driving data to obtain fused data; The determining unit is configured to determine, based on the fused data, the control instructions for the in-vehicle virtual object to be displayed in the vehicle, wherein the control instructions are used to characterize the display rules of the skeletal movements of the in-vehicle virtual object; The second acquisition unit is used to acquire multiple skeletal animations that match the control command; A control unit is used to control the skeletal movements of the in-vehicle virtual object according to the plurality of skeletal animations and the control commands.

9. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method of any one of claims 1 to 7.

Images