Self-driving tour vlog generation system and method based on multi-modal emotion association

Abstract

The application discloses a self-driving tour Vlog generation system and method based on multi-modal emotion correlation, relates to the field of intelligent multimedia technology, and has the technical scheme as follows: the method synchronously collects vehicle ambient environment data, vehicle control data, biological feature data and voice interaction data through a vehicle-mounted system, and constructs a heterogeneous graph neural network model; a multi-modal emotion analysis is used to generate a second-level emotion intensity curve, and scene dynamic characteristics, user preference parameters and a time attenuation factor are combined to calculate a segment comprehensive weight; a candidate segment set is selected according to the comprehensive weight; and a final self-driving tour Vlog is automatically generated by synthesizing a clip structure according to the candidate segment set and the emotion intensity curve. The application breaks through the limitations of the traditional scheme, such as single data dimension and insufficient emotion correlation, realizes a technical leap from'scene recording' to 'experience restoration', and significantly improves the immersion and personalization level of the Vlog.

Description

Technical Field

[0001] This invention relates to the field of intelligent multimedia technology, and more specifically, to a system and method for generating self-driving tour vlogs based on multimodal emotional association. Background Technology

[0002] Existing in-vehicle video generation technologies have significant limitations, with most solutions relying solely on the acquisition of single image data to generate videos. For example, the published Chinese patent CN119996790A, concerning a method, device, vehicle, and computer program for generating car vlog videos, while capable of acquiring environmental images and identifying key events during the journey through in-vehicle sensors, limits its data dimension to visual information. It fails to capture the emotional interaction between passengers and the vehicle during road trips, nor can it correlate vehicle performance data with the travel experience. Consequently, the generated videos merely mechanically showcase the scenery, failing to capture the dynamic driving experience and the passengers' genuine emotional fluctuations, thus failing to meet users' needs for travel records that are "warm and detailed."

[0003] Other related in-vehicle video generation technologies have significant shortcomings. Some solutions rely on multi-vehicle collaboration to solve the problem of image occlusion during driving, while others only achieve the basic function of "editing images in chronological order." Firstly, the data dimensions are limited, only collecting environmental image data and lacking vehicle control data and passenger biometric data, making it difficult to construct a three-dimensional and complete travel memory. Secondly, emotional connection is lacking; no mapping mechanism between video content and passenger emotions has been established, and the selection of background music and editing rhythm are fixed, failing to match emotional changes during the journey. Thirdly, personalization is insufficient; the generation logic relies on preset scene rules and cannot customize content according to user preferences. Fourthly, multimodal fusion is lacking; deep correlation analysis of images, vehicle data, biosignals, and voice has not been achieved, resulting in a lack of depth in video narrative and preventing viewers from perceiving the intrinsic connection between "driving behavior, environmental changes, and passenger reactions."

[0004] In summary, existing in-vehicle video generation technologies generally remain at the superficial level of "recording scenes" and have failed to achieve the core breakthrough of "restoring the experience": they cannot present the dynamic characteristics and emotional interactions of passengers during the driving process through multi-dimensional data fusion, making it difficult to generate self-driving tour vlogs with high commemorative value and immersive experience.

[0005] Therefore, in order to solve the above-mentioned technical problems, this invention proposes a self-driving tour Vlog generation system and method based on multimodal emotional association. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a self-driving tour Vlog generation system and method based on multimodal emotional association.

[0007] To achieve the above objectives, in a first aspect, the present invention provides the following technical solution: a method for generating self-driving tour Vlogs based on multimodal emotion association, the method comprising:

[0008] During the self-driving tour, the vehicle system collects real-time data on the vehicle's surrounding environment, vehicle control data, biometric data, and voice interaction data. All collected data are timestamped based on the same clock source, forming a multimodal data stream for initial spatiotemporal screening.

[0009] After preprocessing the multimodal data stream, it is input into a heterogeneous graph neural network model. The heterogeneous graph neural network model uses the smallest data unit of the multimodal data stream at continuous time nodes as nodes in the graph, and uses the temporal and semantic associations between nodes as edges to perform feature embedding and association strength learning, and outputs a sentiment feature representation that integrates multimodal contextual information.

[0010] Based on the aforementioned emotional feature representation, a time series model is used to generate an emotional intensity curve that reflects the emotional fluctuations throughout the self-driving tour at a resolution of seconds.

[0011] The emotional intensity curve, scene dynamic features extracted from vehicle control data, and user-preset preference parameters are combined with the time decay factor as input to calculate the comprehensive weight of each video segment, and a candidate segment set is selected based on the comprehensive weight.

[0012] Based on the candidate segment set and the emotion intensity curve, the editing structure is automatically generated and the final self-driving tour Vlo is synthesized.

[0013] Furthermore, the vehicle surrounding environment data includes roadside landscape images with geographic coordinates and dynamic traffic participant information collected by surround-view cameras.

[0014] The vehicle control data includes real-time longitudinal acceleration, lateral G-value, steering angular velocity obtained via the vehicle CAN bus, and road gradient and road surface type provided by the navigation system;

[0015] The biometric data includes facial motion unit encoding sequences captured by an infrared camera and skin conductance response signals acquired by a biosensor integrated into the steering wheel.

[0016] The voice interaction data includes the voice text content recorded by the in-vehicle microphone array and the corresponding fundamental frequency and speech rate acoustic features.

[0017] Furthermore, the multimodal data stream is preprocessed, specifically including:

[0018] The timestamps of the multimodal data stream are recalibrated with millisecond-level precision based on a high-precision hardware clock signal;

[0019] Sliding window smoothing filtering and outlier removal based on the isolated forest algorithm are applied to vehicle control data and biometric data.

[0020] The data from each modality are sampled at a fixed frequency of 10 frames per second and normalized to a tensor format within the same numerical range.

[0021] Furthermore, the heterogeneous graph neural network model performs feature embedding and association strength learning, including:

[0022] The initial features of the nodes of the smallest data unit are set, wherein the initial features of the vehicle surrounding environment data nodes are obtained by extracting visual feature vectors through a pre-trained convolutional neural network, the initial features of the vehicle control data nodes are spliced ​​vectors of multi-parameter sampled values, the initial features of the biometric data nodes are spliced ​​vectors of facial action unit encoding sequence and skin conductance response signal, and the initial features of the voice interaction data nodes are spliced ​​vectors of the mean of word vector sequence of voice text content and acoustic features.

[0023] Based on the initial features of the nodes, a heterogeneous graph structure is constructed, including the construction of temporal association edges and semantic association edges. The temporal association edges connect all heterogeneous nodes under the same timestamp, and the semantic association edges calculate the cosine similarity of the initial features of nodes of different modalities based on a pre-trained semantic model and connect them based on a preset threshold.

[0024] The constructed heterogeneous graph structure is input into the graph attention network. The neighbor node information is aggregated through multiple rounds of message passing to update the feature representation of each node. The dynamic weight of each semantic association edge is learned simultaneously as the association strength of the semantic association edge. The association strength and the updated node feature representation are used together for the calculation of sentiment feature representation.

[0025] Furthermore, the emotional feature representation is calculated using the following formula:

[0026] ;

[0027] In the formula, Indicates time The expression of emotional characteristics; This indicates the L2 normalization operation; The number of modes in a multimodal data stream; Indicates the first The weight coefficients for each mode are determined by the modality. The average strength of the semantic association edges connecting nodes; Indicates the updated number The average pooling vector representing the node feature representations of each modality; The fusion coefficient representing contextual information is a preset constant. The number of dimensions representing the context features; Indicates the first Weight coefficients of contextual features; Indicates time The Dimensional context information.

[0028] Furthermore, based on the aforementioned emotional feature representation, the method for generating emotional intensity curves through time series modeling using an emotional analysis model includes:

[0029] For each moment Emotional characteristics Construct a multi-scale feature window that includes recent, medium, and long-term context, where the recent window is... The medium-term window is The long-term window is ;

[0030] The sentiment features within the multi-scale feature windows are input into the corresponding temporal convolutional networks for feature extraction. The recent window uses a one-dimensional convolution with a kernel size of 3, the mid-term window uses a one-dimensional convolution with a kernel size of 5, and the long-term window uses a one-dimensional convolution with a kernel size of 7. The extracted features are then concatenated to form multi-scale temporal features.

[0031] The multi-scale temporal features and the context condition vector composed of emergency braking and sharp turning event markers extracted from vehicle control data are jointly input into a gated recurrent unit network. Through nonlinear transformation, the input multi-scale temporal features and context condition vector are mapped into scalar form of sentiment intensity. ;

[0032] All moments of the self-driving tour Calculated emotional intensity Sort and align according to the corresponding timestamps to form the initial discrete emotional intensity time series;

[0033] The initial discrete emotional intensity time series is subjected to scene-adaptive smoothing optimization. While retaining the sharp emotional peaks triggered by sudden braking and sharp turning events, irrelevant high-frequency jitter caused by road bumps, constant speed cruising, traffic light waiting and sensor noise is filtered out, and the final emotional intensity curve is output.

[0034] Furthermore, the formula for calculating the overall weight of each video segment is as follows:

[0035] ;

[0036] In the formula, For the first The overall weight of each video segment; The preset weighting coefficients, and ; For the first The emotional intensity value of each video segment is taken as the average emotional intensity value of the corresponding emotional intensity curve of that video segment; For the first The scene dynamic feature value of a video segment is taken as the average of the sum of the absolute value of the vehicle's lateral G-value and the absolute value of the real-time longitudinal acceleration within that video segment; For the first The user preference parameter matching value for a video segment is the cosine similarity between the average vector of all emotional feature representations in the video segment and the user's preset preference parameters. For the first The time decay factor of each video segment ,in For the attenuation rate parameter, For the first The start time of each video segment This represents the total duration of the journey.

[0037] Furthermore, a set of candidate segments is selected based on the aforementioned comprehensive weights, specifically including:

[0038] Based on the comprehensive weight of all video segments, a peak detection algorithm based on kernel density estimation is used to determine the dynamic screening threshold, and video segments with a comprehensive weight greater than the dynamic screening threshold are selected as initial candidate segments.

[0039] Based on the start and end times of the initial candidate segments, adaptive expansion is performed in the forward and backward directions of the time sequence. If adjacent segments simultaneously meet the following three conditions:

[0040] Adjacent segments are temporally continuous with the initial candidate segments;

[0041] The slope sign of the emotion intensity curves of adjacent segments is consistent with that of the initial candidate segments;

[0042] The combined weight of adjacent segments shall not be less than 70% and not more than 130% of the dynamic screening threshold;

[0043] Then the adjacent segment will be merged into the candidate segment;

[0044] Optimize the total duration of the expanded candidate segments: if the total duration Exceeding the preset limit Then, according to the overall weight in ascending order, the candidate segments with the lowest weight are iteratively removed until the total duration meets the requirement. If the total duration Below the preset lower limit Then, based on the peak density of the emotion intensity curve, non-candidate segments with high weights are added from high-density regions until the total duration meets the requirement. ;

[0045] The final output is a set of candidate segments.

[0046] Furthermore, based on the candidate segment set and the emotion intensity curve, an editing structure is automatically generated and the final road trip Vlog is synthesized, including:

[0047] Based on a preset emotional intensity-editing rule mapping library, an editing mode is assigned to each segment in the candidate segment set. When the slope of the emotional intensity curve in the segment is continuously positive, an editing mode in which the shot duration decreases as the emotional intensity increases is assigned. When the segment contains a local maximum value of the emotional intensity curve, a fast-paced combination editing mode in which multi-view material alignment and duration compression are performed based on the peak point is assigned.

[0048] Based on the difference in emotional intensity between adjacent segments in the candidate segment set, the corresponding video transition type and audio transition effect are automatically matched. When the difference in emotional intensity exceeds a preset positive threshold, a fast-switching transition effect is matched with a sound effect selected from a preset high-frequency sound effect library. When the difference in emotional intensity is negative, a fade-in / fade-out transition effect is matched with a sound effect selected from a preset low-frequency sound effect library.

[0049] Based on a unified timestamp, the vehicle surrounding environment data, biometric data corresponding to changes in emotional intensity, and voice interaction data in the candidate segment set are synchronously synthesized, and the final self-driving tour Vlog is generated through picture-in-picture, split-screen, or time-alternating layout methods.

[0050] Secondly, based on the method of generating self-driving tour Vlogs based on multimodal emotional association, a generation system for generating self-driving tour Vlogs based on multimodal emotional association is provided. This system includes a multimodal data acquisition module, which is used to collect real-time data on the vehicle's surrounding environment, vehicle control data, biometric data, and voice interaction data through the in-vehicle system during the self-driving tour. All data are timestamped based on the same clock source to form a multimodal data stream with initial spatiotemporal screening.

[0051] The data association and feature learning module is used to preprocess the multimodal data stream and input it into the heterogeneous graph neural network model. It uses the smallest data unit at continuous time nodes as nodes and time association and semantic association as edges to perform feature embedding and association strength learning, and outputs a sentiment feature representation that integrates multimodal context information.

[0052] The sentiment analysis module is used to perform time series modeling based on the sentiment feature representation and the sentiment analysis model to generate a sentiment intensity curve that reflects the emotional fluctuations throughout the self-driving tour at a resolution of seconds.

[0053] The intelligent editing decision module is used to take the emotional intensity curve, the scene dynamic features extracted from vehicle control data, the user's preset preference parameters and time decay factor as input, calculate the comprehensive weight of each video segment, and filter out a set of candidate segments based on the comprehensive weight.

[0054] The Vlog synthesis output module is used to automatically generate an editing structure and synthesize the final road trip Vlog based on the candidate segment set and the emotion intensity curve.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] By integrating in-vehicle surround view images, CAN bus vehicle control data, biometric data, and voice interaction data, a heterogeneous graph neural network model is constructed, breaking through the limitations of traditional solutions with their single data dimension. Innovatively, multimodal sentiment analysis is used to generate second-level sentiment intensity curves, driving dynamic matching of editing rhythm and background music style to achieve audio-visual emotional synchronization. Combining user preferences and automotive scene characteristics, a comprehensive weight model including sentiment intensity, scene dynamics, user preferences, and time decay factors enables personalized segment selection. Ultimately, a multi-platform adaptive Vlog is generated that combines driving experience recreation with emotional memory presentation, solving the problems of lack of emotional connection and insufficient personalization in existing technologies, and completing a technological leap from "scene recording" to "experience recreation." Attached Figure Description

[0057] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0058] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0060] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0061] like Figure 1 As shown, this is an embodiment of the present invention, which provides a method for generating self-driving tour Vlogs based on multimodal emotion association.

[0062] Before embarking on a road trip, users can preset their preferences via the in-vehicle terminal or mobile app. The system offers several preset templates, including "Family," "Adventure," and "Sightseeing." The "Family" version prioritizes close-ups of children's expressions and family interactions, the "Adventure" version focuses on vehicle performance data and intense driving scenarios, and the "Sightseeing" version emphasizes scenery and attraction descriptions along the way. Users can fine-tune parameters, including setting "Driving Data Display Intensity" (0-100%), "Children's Perspective Priority" (Yes / No), and "Music Style Preference" (Exciting / Soothing / Auto). The system also supports implicit preference learning based on historical behavior, automatically calculating the weights of tags such as "Driving Enthusiast" and "Landscape Photography" by analyzing previously saved, viewed, or shared Vlog clips.

[0063] The method includes the following steps:

[0064] S1: During the self-driving tour, the vehicle system collects real-time data on the vehicle's surrounding environment, vehicle control data, biometric data, and voice interaction data. All collected data are timestamped based on the same clock source, forming a multimodal data stream for initial spatiotemporal screening.

[0065] During the self-driving tour, the system collects multimodal data in real time through the onboard hardware system. Surround environment data is collected by a surround-view system consisting of four 200-degree wide-angle cameras installed around the vehicle, simultaneously recording latitude and longitude coordinates, altitude, and precise time information provided by the GPS module. The surround-view system not only captures images of the surrounding landscape but also uses the YOLOv5 model to detect dynamic traffic participants in real time, including vehicles, pedestrians, and animals, and marks their relative positions and movement trajectories. Vehicle control data is collected via the onboard CAN bus at a frequency of 100Hz, specifically including: longitudinal acceleration, lateral G-value, steering angular velocity, and road gradient and road surface type codes provided by the navigation system (1=asphalt, 2=cement, 3=gravel, 4=mud). CAN bus data is read in real time through the OBD-II interface, with each data point accompanied by a timestamp accurate to the millisecond. Biometric data is collected from two sources: an infrared camera mounted on the dashboard captures facial images of the driver and front passenger at 30fps, extracting 17 facial motion unit encoding sequences using the OpenFace toolkit, including key features such as eye opening, mouth corner angle, and frowning degree; a biosensor integrated into the steering wheel collects the driver's skin conductance response signal at a frequency of 10Hz, with a measurement range of 0-100. Resolution 0.1 It can also be collected through wearable devices. Voice interaction data is collected through a 4-microphone array deployed on the roof of the vehicle, with a sampling rate of 16kHz and a bit depth of 16bit. The voice data undergoes real-time voice activity detection and speaker separation, is converted into text by an ASR engine, and acoustic features such as fundamental frequency, speech rate (syllables / second), and energy (dB) are extracted.

[0066] All collected data uses the vehicle's central gateway clock as a unified reference, with an accuracy of ±1ms. The NTP protocol is used to correct the timestamps of external devices such as GPS (accuracy ±1s) and wearable devices (accuracy ±1s) to ensure all data is aligned to the "vehicle clock coordinate system," with a maximum time error not exceeding 10ms. The specific time synchronization process includes: first, calibrating the clocks of all external devices using the NTP protocol with the vehicle's central gateway clock as the reference; then, recalibrating the timestamp of each data point with millisecond-level accuracy; and finally, using a time interpolation algorithm to ensure all modal data are aligned on a unified time axis. For example, the "lateral G value = 0.9" (10:23:45.123) collected by the CAN bus, the "30°15' North Latitude" (10:23:45) from GPS, and the "corner of the mouth upturned 0.8" (10:23:45.120) from the infrared camera are all uniformly marked as "10:23:45.123" after calibration.

[0067] S2: After preprocessing the multimodal data stream, it is input into the heterogeneous graph neural network model; the heterogeneous graph neural network model uses the smallest data unit of the multimodal data stream at continuous time nodes as nodes in the graph, and uses the temporal and semantic associations between nodes as edges to perform feature embedding and association strength learning, and outputs a sentiment feature representation that integrates multimodal contextual information.

[0068] The preprocessing of the multimodal data stream includes three steps: First, the timestamps are recalibrated with millisecond-level precision based on a high-precision hardware clock signal; then, sliding window smoothing filtering and outlier removal based on the isolated forest algorithm are applied to vehicle control data and biometric data; finally, all modal data are uniformly sampled to a fixed frequency of 10 frames per second and normalized to a tensor format within the same numerical range. In the smoothing filtering stage, a Savitzky-Golay filter (polynomial order 2, window length 11) is used to smooth the data, preserving the true signal trend while suppressing high-frequency noise. In the outlier detection stage, the isolated forest algorithm is used, with a contamination parameter set to 0.1, to remove abnormal data points caused by sensor malfunctions or external interference. In the resampling stage, for source data with different sampling rates, vehicle control data (100Hz) is downsampled, while biometric data (30Hz) and voice data (after 16kHz feature extraction) are upsampled, all using linear interpolation. Finally, all features are normalized to the [0,1] interval and organized into a tensor format with uniform dimensions.

[0069] The preprocessed multimodal data stream is input into a heterogeneous graph neural network model. The model uses the smallest data unit of the multimodal data stream at continuous time nodes as nodes in the graph. The initial features of the nodes are set as follows: the initial features of the vehicle surrounding environment data nodes are obtained by extracting a 1024-dimensional visual feature vector through a pre-trained convolutional neural network (using the MobileNetV3 architecture); the initial features of the vehicle control data nodes are a concatenated vector of multi-parameter sampled values ​​(including 8 parameters such as longitudinal acceleration, lateral G value, and steering angular velocity); the initial features of the biometric data nodes are a concatenated vector of facial action unit encoding sequence (17-dimensional) and skin conductance response signal (1-dimensional); the initial features of the voice interaction data nodes are a concatenated vector of the mean of the word vector sequence of the voice text content (extracted using BERT pre-trained in the automotive scene, dimension 768) and acoustic features (fundamental frequency, speech rate, energy, 3-dimensional). The construction of edges in the graph includes temporally related edges and semantically related edges: temporally related edges connect all heterogeneous nodes under the same timestamp, with a fixed edge weight of 1; semantically related edges calculate the cosine similarity of the initial features of nodes of different modalities based on a pre-trained semantic model (using Sentence-BERT), and connect them based on a preset threshold of 0.7, with edge weights initialized to the similarity values. The constructed heterogeneous graph structure is input into a graph attention network, which contains a 3-layer graph attention mechanism, with 8 attention heads in each layer. Neighbor node information is aggregated through multiple rounds of message passing to update the feature representation of each node, and the dynamic weight of each semantically related edge is learned synchronously as the association strength of that semantically related edge. Sentiment feature representation is calculated using the following formula:

[0070] ;

[0071] In the formula, Indicates time The expression of emotional characteristics; This indicates the L2 normalization operation; The number of modes in a multimodal data stream; Indicates the first The weight coefficients for each mode are determined by the modality. The average strength of the semantic association edges connecting nodes; Indicates the updated number The average pooling vector representing the node feature representations of each modality; The fusion coefficient representing contextual information is a preset constant. The number of dimensions representing the context features; Indicates the first Weight coefficients of contextual features; Indicates time The Dimensional context information.

[0072] During the process of inputting the constructed heterogeneous graph structure into the graph attention network for feature updating, the dynamic weights of semantically related edges are calculated and updated through the graph attention mechanism. Specifically, in each layer of the graph attention network, for any semantically related edge, based on the node feature representations of the two heterogeneous nodes connected by the semantically related edge in the current layer, combined with the initial weights of the semantically related edge, the corresponding attention coefficient is calculated by inputting it into the attention function; wherein, the attention coefficient is used to characterize the association strength of the semantically related edge in the information transmission process of the current layer.

[0073] The attention coefficients are obtained by normalizing the output of the attention function and participate in the weighted aggregation of neighbor node features as the dynamic weights of the semantically related edge in the current layer. In the multi-layer graph attention network structure, the dynamic weights of the semantically related edge are recalculated in each layer according to the node features of the corresponding layer, thereby achieving dynamic updates with network layers and the training process.

[0074] During model training, the error between the predicted result of the sentiment feature representation and the corresponding real sentiment label is used as the optimization objective. The parameters in the graph attention network are updated through the backpropagation algorithm, so that the dynamic weights of the semantic association edges are adaptively adjusted during training to reflect the contribution of the semantic association strength between nodes of different modalities to the sentiment feature representation.

[0075] S3: Based on the representation of emotional features, time series modeling is performed using an emotional analysis model to generate an emotional intensity curve that reflects the emotional fluctuations throughout the self-driving tour at a resolution of seconds.

[0076] The specific methods include: for each time step Emotional characteristics Construct a multi-scale feature window that includes recent, medium, and long-term context, where the recent window is... (3 seconds), the mid-term window is 6 seconds), long-term window is (11 seconds). The sentiment feature representations within each scale window are input into the corresponding temporal convolutional networks for feature extraction. The recent window uses a one-dimensional convolution with a kernel size of 3 (64 kernels), the mid-term window uses a one-dimensional convolution with a kernel size of 5 (128 kernels), and the long-term window uses a one-dimensional convolution with a kernel size of 7 (256 kernels). The extracted features are concatenated to form a 448-dimensional multi-scale temporal feature. This multi-scale temporal feature, along with a contextual condition vector (4-dimensional) composed of emergency braking and sharp turning event markers extracted from vehicle control data, is input into a gated recurrent unit (GRU) network. The network contains 3 layers of GRUs, each with 256 hidden units and a dropout rate of 0.2. A nonlinear transformation maps the input temporal features and the contextual condition vector into scalar forms of sentiment intensity. Using the Sigmoid activation function ensures the output range is [0, 10]. This will capture all moments throughout the entire road trip. Calculated emotional intensity The data is sorted and aligned according to the corresponding timestamps to form an initial discrete emotional intensity time series. A scene-adaptive smoothing optimization is then performed on this initial discrete emotional intensity time series. While retaining sharp emotional peaks triggered by events such as sudden braking (longitudinal acceleration <-3m / s²) and sharp turns (lateral G-value >0.8g), irrelevant high-frequency jitter caused by road bumps (longitudinal acceleration variance >2), constant speed cruising (absolute acceleration <0.5m / s² for 10 seconds), waiting at traffic lights (vehicle speed = 0 for 30 seconds), and sensor noise is filtered out. Specifically, Gaussian filtering (σ=1.5) is used for smoothing, outputting the final emotional intensity curve.

[0077] S4: The emotional intensity curve, scene dynamic features extracted from vehicle control data, and user-preset preference parameters are combined with the time decay factor as input to calculate the comprehensive weight of each video segment, and a candidate segment set is selected based on the comprehensive weight.

[0078] Specifically, the scene dynamic feature is taken as the average of the sum of the absolute values ​​of the vehicle's lateral G-value and the absolute values ​​of its real-time longitudinal acceleration within the video clip; the user preference parameter matching value is taken as the cosine similarity between the average vector of all emotional feature representations within the video clip and the user's preset preference parameters; the time decay factor is calculated using the following formula: ,in For the attenuation rate parameter, For the first The start time of each video segment This represents the total journey time. The formula for calculating the overall weight is:

[0079] ;

[0080] In the formula, For the first The overall weight of each video segment; The preset weighting coefficients, and ; For the first The emotional intensity value of each video segment is taken as the average emotional intensity value of the corresponding emotional intensity curve of that video segment; For the first The scene dynamic feature value of a video segment is taken as the average of the sum of the absolute value of the vehicle's lateral G-value and the absolute value of the real-time longitudinal acceleration within that video segment; For the first The user preference parameter matching value for a video segment is the cosine similarity between the average vector of all emotional feature representations in the video segment and the user's preset preference parameters. For the first The time decay factor of each video segment.

[0081] The weighting coefficients are dynamically adjusted based on the current scenario: settings for driving scenarios (such as turning on a mountain road). The values ​​are 0.3+0.35+0.25+0.1 respectively; 0.3+0.25+0.35+0.1 for family scenarios (such as children's interaction); and 0.4+0.2+0.3+0.1 for emotional scenarios (such as first impressions at scenic spots).

[0082] All input parameters need to be normalized to a unified dimension. (Emotional intensity value) Linearly scale it to the [0,1] interval; scene dynamic feature values As a dimensional physical quantity, based on a large amount of self-driving tour data statistics, its value range is mapped to [0,1], and normalized using a piecewise linear function; user preference parameter matching value Since it is a cosine similarity, its value range naturally falls in the interval [-1, 1]. To preserve the positive values ​​of all parameters, it is linearly transformed to [0, 1]; time decay factor It is itself a dimensionless scalar in the interval [0,1], and no additional processing is required.

[0083] A candidate segment set is selected based on a comprehensive weighting, specifically including: determining a dynamic screening threshold using a peak detection algorithm based on kernel density estimation, based on the comprehensive weighting of all video segments. Kernel density estimation uses a Gaussian kernel function with a bandwidth of 0.5, using the local maximum point of the density function as the candidate center, and selecting video segments with a comprehensive weight greater than the dynamic screening threshold as initial candidate segments. Using the start and end times of the initial candidate segments as a baseline, adaptive expansion is performed in the temporal direction. If adjacent segments simultaneously meet the following three conditions: the adjacent segment is temporally continuous with the initial candidate segment (interval less than 2 seconds); the slope sign of the emotional intensity curve of the adjacent segment is consistent with that of the initial candidate segment; and the comprehensive weight of the adjacent segment is not less than 70% and not more than 130% of the dynamic screening threshold, then the adjacent segment is merged into the candidate segment set. The total duration of the expanded candidate segments is optimized: if the total duration exceeds a preset upper limit... (180 seconds) Then, according to the comprehensive weight in ascending order, the candidate segments with the lowest weight are iteratively removed until the total duration satisfies T≤ If the total duration is lower than the preset lower limit (120 seconds) Then, according to the peak density of the emotion intensity curve, non-candidate segments with high weights are added from high-density regions until the total duration meets the requirement. The final output is a set of candidate fragments.

[0084] S5: Based on the candidate segment set and emotional intensity curve, automatically generate the editing structure and synthesize the final road trip Vlog.

[0085] Specifically, this includes: assigning editing modes to each segment in the candidate segment set based on a preset emotion intensity-editing rule mapping library. When the slope of the emotion intensity curve within a segment is consistently positive, an editing mode is assigned where the shot duration decreases as the emotion intensity increases, starting with a 2-second shot duration and decreasing geometrically by 0.8. When a segment contains a local maximum value of the emotion intensity curve, a fast-paced combination editing mode is assigned, which aligns multi-view footage and compresses duration based on the peak point, aligning multi-view footage within 0.5 seconds before and after the peak point and compressing duration (compression ratio 0.3). Based on the difference in emotion intensity between adjacent segments in the candidate segment set, corresponding video transition types and audio transition effects are automatically matched. When the difference in emotion intensity exceeds a preset positive threshold, a fast-switching transition effect is matched with sound effects selected from a preset high-quality sound effects library (containing 200 sound effects); when the difference in emotion intensity is negative, a fade-in / fade-out transition effect is matched with sound effects selected from a preset low-quality sound effects library (containing 150 sound effects). Based on a unified timestamp, the vehicle's surrounding environment data, biometric data corresponding to changes in emotional intensity (such as heart rate curves and facial expression charts), and voice interaction data from the candidate segment set are synchronously synthesized. The final self-driving tour Vlog is generated through layout methods such as picture-in-picture (the main screen is the environmental image, and the small window displays biometrics), split screen (left and right split screens display driving data and scenery), or time-series alternation (the display focus is switched according to the emotional intensity).

[0086] This embodiment also discloses a generation system based on the above method, the system comprising:

[0087] The multimodal data acquisition module is used to collect real-time data on the vehicle's surrounding environment, vehicle control data, biometric data, and voice interaction data through the in-vehicle system during self-driving tours. It also adds timestamps to all data based on the same clock source, forming a multimodal data stream with initial spatiotemporal screening.

[0088] The data association and feature learning module is used to preprocess multimodal data streams and input them into a heterogeneous graph neural network model. It uses the smallest data unit at continuous time nodes as nodes and time association and semantic association as edges to perform feature embedding and association strength learning, and outputs sentiment feature representations that integrate multimodal contextual information.

[0089] The sentiment analysis module is used to perform time series modeling based on sentiment feature representation and sentiment analysis model to generate sentiment intensity curves that reflect the emotional fluctuations throughout the self-driving tour at a resolution of seconds.

[0090] The intelligent editing decision module takes the emotional intensity curve, scene dynamic features extracted from vehicle control data, user-preset preference parameters, and time decay factor as inputs to calculate the comprehensive weight of each video segment and select a set of candidate segments based on the comprehensive weight.

[0091] The Vlog synthesis and output module automatically generates an editing structure and synthesizes the final road trip Vlog based on a set of candidate clips and an emotional intensity curve. All modules in the system work together to achieve a fully automated process from data acquisition to Vlog generation.

[0092] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Those skilled in the art can readily implement the present invention based on the accompanying drawings and the above description. However, any modifications, alterations, or variations made by those skilled in the art without departing from the scope of the present invention, utilizing the disclosed technical content, are equivalent embodiments of the present invention. Furthermore, any modifications, alterations, or variations made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the present invention.

Claims

1. A method for generating self-driving tour vlogs based on multimodal sentiment association, characterized in that, The method includes: During the self-driving tour, the vehicle system collects real-time data on the vehicle's surrounding environment, vehicle control data, biometric data, and voice interaction data. All collected data are timestamped based on the same clock source, forming a multimodal data stream for initial spatiotemporal screening. After preprocessing the multimodal data stream, it is input into a heterogeneous graph neural network model. The heterogeneous graph neural network model uses the smallest data unit of the multimodal data stream at continuous time nodes as nodes in the graph, takes the temporal and semantic associations between nodes as edges, and models the semantic associations through a graph attention mechanism. It synchronously learns the dynamic weights of the semantic association edges as the association strength, performs feature embedding and association strength learning, and outputs a sentiment feature representation that integrates multimodal contextual information. Based on the aforementioned emotional feature representation, a time series model is used to generate an emotional intensity curve that reflects the emotional fluctuations throughout the self-driving tour at a resolution of seconds. The emotional intensity curve, scene dynamic features extracted from vehicle control data, and user-preset preference parameters are combined with the time decay factor as input to calculate the comprehensive weight of each video segment, and a candidate segment set is selected based on the comprehensive weight. The formula for calculating the overall weight of each video segment is as follows: ； In the formula, For the first The overall weight of each video segment; The preset weighting coefficients, and ; For the first The emotional intensity value of each video segment is taken as the average emotional intensity value of the corresponding emotional intensity curve of that video segment; For the first The scene dynamic feature value of a video segment is taken as the average of the sum of the absolute value of the vehicle's lateral G-value and the absolute value of the real-time longitudinal acceleration within that video segment; For the first The user preference parameter matching value for a video segment is the cosine similarity between the average vector of all emotional feature representations in the video segment and the user's preset preference parameters. For the first The time decay factor of each video segment ,in For the attenuation rate parameter, For the first The start time of each video segment Total journey duration; Based on the candidate segment set and the emotion intensity curve, the editing structure is automatically generated and the final road trip Vlog is synthesized.

2. The method for generating self-driving tour Vlogs based on multimodal sentiment association according to claim 1, characterized in that, The vehicle surrounding environment data includes roadside landscape images with geographic coordinates and dynamic traffic participant information collected by surround-view cameras. The vehicle control data includes real-time longitudinal acceleration, lateral G-value, steering angular velocity obtained via the vehicle CAN bus, and road gradient and road surface type provided by the navigation system; The biometric data includes facial motion unit encoding sequences captured by an infrared camera and skin conductance response signals acquired by a biosensor integrated into the steering wheel. The voice interaction data includes the voice text content recorded by the in-vehicle microphone array and the corresponding fundamental frequency and speech rate acoustic features.

3. The method for generating self-driving tour Vlogs based on multimodal emotion association according to claim 2, characterized in that, Preprocessing the multimodal data stream specifically includes: The timestamps of the multimodal data stream are recalibrated with millisecond-level precision based on a high-precision hardware clock signal; Sliding window smoothing filtering and outlier removal based on the isolated forest algorithm are applied to vehicle control data and biometric data. The data from each modality are sampled at a fixed frequency of 10 frames per second and normalized to a tensor format within the same numerical range.

4. The method for generating self-driving tour Vlogs based on multimodal emotion association according to claim 3, characterized in that, The heterogeneous graph neural network model performs feature embedding and association strength learning, including: The initial features of the nodes of the smallest data unit are set, wherein the initial features of the vehicle surrounding environment data nodes are obtained by extracting visual feature vectors through a pre-trained convolutional neural network, the initial features of the vehicle control data nodes are spliced ​​vectors of multi-parameter sampled values, the initial features of the biometric data nodes are spliced ​​vectors of facial action unit encoding sequence and skin conductance response signal, and the initial features of the voice interaction data nodes are spliced ​​vectors of the mean of word vector sequence of voice text content and acoustic features. Based on the initial features of the nodes, a heterogeneous graph structure is constructed, including the construction of temporal association edges and semantic association edges. The temporal association edges connect all heterogeneous nodes under the same timestamp, and the semantic association edges calculate the cosine similarity of the initial features of nodes of different modalities based on a pre-trained semantic model and connect them based on a preset threshold. The constructed heterogeneous graph structure is input into the graph attention network. The neighbor node information is aggregated through multiple rounds of message passing to update the feature representation of each node. The dynamic weight of each semantic association edge is learned simultaneously as the association strength of the semantic association edge. The association strength and the updated node feature representation are used together for the calculation of sentiment feature representation.

5. The method for generating self-driving tour Vlogs based on multimodal sentiment association according to claim 4, characterized in that, The emotional characteristics are calculated using the following formula: ； In the formula, Indicates time The expression of emotional characteristics; This represents the L2 normalization operation; The number of modes in a multimodal data stream; Indicates the first The weight coefficients for each mode are determined by the modality. The average strength of the semantic association edges connecting nodes; Indicates the updated number The average pooling vector representing the node feature representations of each modality; The fusion coefficient representing contextual information is a preset constant. The number of dimensions representing the context features; Indicates the first Weight coefficients of contextual features; Indicates time The Dimensional context information.

6. The method for generating self-driving tour Vlogs based on multimodal emotion association according to claim 5, characterized in that, Based on the aforementioned sentiment feature representation, methods for generating sentiment intensity curves through time series modeling using sentiment analysis models include: For each moment Emotional characteristics Construct a multi-scale feature window that includes recent, medium, and long-term context, where the recent window is... The medium-term window is The long-term window is ; The sentiment features within the multi-scale feature windows are input into the corresponding temporal convolutional networks for feature extraction. The recent window uses a one-dimensional convolution with a kernel size of 3, the mid-term window uses a one-dimensional convolution with a kernel size of 5, and the long-term window uses a one-dimensional convolution with a kernel size of 7. The extracted features are then concatenated to form multi-scale temporal features. The multi-scale temporal features and the context condition vector composed of emergency braking and sharp turning event markers extracted from vehicle control data are jointly input into a gated recurrent unit network. Through nonlinear transformation, the input multi-scale temporal features and context condition vector are mapped into scalar form of sentiment intensity. ; All moments of the self-driving tour Calculated emotional intensity Sort and align according to the corresponding timestamps to form the initial discrete emotional intensity time series; The initial discrete emotional intensity time series is subjected to scene-adaptive smoothing optimization. While retaining the sharp emotional peaks triggered by sudden braking and sharp turning events, irrelevant high-frequency jitter caused by road bumps, constant speed cruising, traffic light waiting and sensor noise is filtered out, and the final emotional intensity curve is output.

7. The method for generating self-driving tour Vlogs based on multimodal sentiment association according to claim 1, characterized in that, The candidate fragment set is selected based on the comprehensive weight, specifically including: Based on the comprehensive weight of all video segments, a peak detection algorithm based on kernel density estimation is used to determine the dynamic screening threshold, and video segments with a comprehensive weight greater than the dynamic screening threshold are selected as initial candidate segments. Based on the start and end times of the initial candidate segments, adaptive expansion is performed in the forward and backward directions of the time sequence. If adjacent segments simultaneously meet the following three conditions: Adjacent segments are temporally continuous with the initial candidate segments; The slope sign of the emotion intensity curves of adjacent segments is consistent with that of the initial candidate segments; The combined weight of adjacent segments shall not be less than 70% and not more than 130% of the dynamic screening threshold; Then the adjacent segment will be merged into the candidate segment; Optimize the total duration of the expanded candidate segments: if the total duration Exceeding the preset limit Then, according to the overall weight in ascending order, the candidate segments with the lowest weight are iteratively removed until the total duration meets the requirement. If the total duration Below the preset lower limit Then, based on the peak density of the emotion intensity curve, non-candidate segments with high weights are added from high-density regions until the total duration meets the requirement. ; The final output is a set of candidate segments.

8. The method for generating self-driving tour Vlogs based on multimodal sentiment association according to claim 1, characterized in that, Based on the candidate segment set and the emotion intensity curve, an editing structure is automatically generated and the final road trip Vlog is synthesized, including: Based on a preset emotional intensity-editing rule mapping library, an editing mode is assigned to each segment in the candidate segment set. When the slope of the emotional intensity curve in the segment is continuously positive, an editing mode in which the shot duration decreases as the emotional intensity increases is assigned. When the segment contains a local maximum value of the emotional intensity curve, a fast-paced combination editing mode in which multi-view material alignment and duration compression are performed based on the peak point is assigned. Based on the difference in emotional intensity between adjacent segments in the candidate segment set, the corresponding video transition type and audio transition effect are automatically matched. When the difference in emotional intensity exceeds a preset positive threshold, a fast-switching transition effect is matched with a sound effect selected from a preset high-frequency sound effect library. When the difference in emotional intensity is negative, a fade-in / fade-out transition effect is matched with a sound effect selected from a preset low-frequency sound effect library. Based on a unified timestamp, the vehicle surrounding environment data, biometric data corresponding to changes in emotional intensity, and voice interaction data in the candidate segment set are synchronously synthesized. The final self-driving tour Vlog is generated through picture-in-picture, split-screen, or time-series alternating layout methods.

9. The generation system of the self-driving tour Vlog generation method based on multimodal emotion association as described in any one of claims 1-8, characterized in that, The system includes: The multimodal data acquisition module is used to collect real-time data on the vehicle's surrounding environment, vehicle control data, biometric data, and voice interaction data through the in-vehicle system during self-driving tours. It also adds timestamps to all data based on the same clock source, forming a multimodal data stream with initial spatiotemporal screening. The data association and feature learning module is used to preprocess the multimodal data stream and input it into the heterogeneous graph neural network model. It uses the smallest data unit at continuous time nodes as nodes and time association and semantic association as edges to perform feature embedding and association strength learning, and outputs a sentiment feature representation that integrates multimodal context information. The sentiment analysis module is used to perform time series modeling based on the sentiment feature representation and the sentiment analysis model to generate a sentiment intensity curve that reflects the emotional fluctuations throughout the self-driving tour at a resolution of seconds. The intelligent editing decision module is used to take the emotional intensity curve, the scene dynamic features extracted from vehicle control data, the user's preset preference parameters and time decay factor as input, calculate the comprehensive weight of each video segment, and filter out a set of candidate segments based on the comprehensive weight. The Vlog synthesis output module is used to automatically generate an editing structure and synthesize the final road trip Vlog based on the candidate segment set and the emotion intensity curve.

Images