Method, apparatus and electronic device for processing vehicle member data

Abstract

The application relates to a vehicle member data processing method and device and electronic equipment. The method comprises the following steps: acquiring in-vehicle sensing data corresponding to a target member in a vehicle, and performing feature extraction on the in-vehicle sensing data to obtain first features; acquiring interaction data of the target member in at least one interaction channel, and performing feature extraction on the interaction data to obtain second features; performing feature fusion processing on the first features and the second features to obtain state features of the target member, and determining a member state of the target member based on the state features. The method can improve the accuracy of the obtained member state of the target member, and help to improve the accuracy of personalized active services provided based on the member state.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a method for processing vehicle occupant data, a device for processing vehicle occupant data, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of vehicle technology, the intelligent cockpit environment has emerged, creating a demand for personalized proactive services by acquiring and processing vehicle occupant data. This process involves using one or more sensors installed in the vehicle to perceive the state of the vehicle occupants (driver and / or passengers) within the intelligent cockpit. Feature vectors are extracted from the state data perceived by these sensors, concatenated, and then classified. The classification results are then used to provide personalized proactive services to the respective occupants. As the demand for personalized proactive services in the intelligent cockpit environment increases, there is a need for new methods of processing vehicle occupant data to further improve the accuracy of the obtained state characteristics of the vehicle occupants. Summary of the Invention

[0003] Based on this, this application addresses the aforementioned technical problems by providing a method for processing vehicle occupant data, a device for processing vehicle occupant data, an electronic device, a computer-readable storage medium, and a computer program product that can improve the accuracy of obtained vehicle occupant status characteristics.

[0004] Firstly, this application provides a method for processing vehicle occupant data, the method comprising:

[0005] Acquire in-vehicle perception data corresponding to the target occupants inside the vehicle, and extract features from the in-vehicle perception data to obtain the first feature;

[0006] Acquire the interaction data of the target member on at least one interaction channel, and extract features from the interaction data to obtain a second feature;

[0007] The first and second features are fused to obtain the state features of the target member, and the member state of the target member is determined based on the state features.

[0008] The vehicle occupant data processing method based on the embodiments described above involves obtaining in-vehicle perception data corresponding to a target occupant, extracting features from the in-vehicle perception data to obtain a first feature, and simultaneously acquiring interaction data of the target occupant on at least one interaction channel, extracting features from the interaction data to obtain a second feature, and then performing feature fusion processing on the first and second features to obtain the target occupant's state features. Finally, the occupant's state is determined based on these state features. In other words, during the processing of vehicle occupant data, the global state features obtained from the processing of the vehicle occupant data consider both the target occupant's state within the vehicle (i.e., the in-vehicle perception data corresponding to the target occupant) and the target occupant's interaction data on at least one interaction channel. Since target members typically engage in personalized interactions through interactive channels, their interaction data reflects their individual behavioral characteristics. Therefore, by fusing the first feature derived from in-vehicle perception data and the second feature derived from interaction data, a state feature is obtained. This state feature reflects both the target member's personalized state while in the vehicle and their personalized state while interacting through interactive channels. Using this state feature as a global feature vector to determine the member's state improves the accuracy of the obtained member state. Furthermore, providing personalized proactive services based on this member state in the intelligent cockpit environment also helps improve the accuracy of the provided personalized proactive services.

[0009] In some optional embodiments, the interaction data includes at least one of the following: interaction log data obtained from the target website; interaction feedback data obtained from the target mini-program; and page jump paths obtained from the target client.

[0010] Feature extraction is performed on the interaction data to obtain the second feature, including:

[0011] In response to interaction data, including interaction log data, a user behavior sequence is generated based on the interaction log data, and the user behavior sequence is sequence encoded to obtain a first feature vector;

[0012] In response to interactive data, including interactive feedback data, feature encoding is performed on the interactive feedback data to obtain a second feature vector;

[0013] In response to the interactive data, including the page navigation path, the page navigation path is preprocessed into a directed graph, where the nodes in the directed graph represent business units in the target client, and the edges in the directed graph represent the navigation relationship between two connected business units; a graph neural network is used to encode the directed graph to obtain a third feature vector;

[0014] The second feature is determined based on at least one of the first feature vector, the second feature vector, and the third feature vector.

[0015] Accordingly, the interaction data of the target member on at least one interaction channel may include at least one of the following: interaction log data obtained from the target website, interaction feedback data obtained from the target mini-program, and page jump paths obtained from the target client. The second feature is determined based on at least one of the first feature vector corresponding to the interaction log data, the second feature vector corresponding to the interaction feedback data, and the third feature vector corresponding to the page jump path. This allows for a full integration of the personalized characteristics reflected by the interaction log data, interaction feedback data, and page jump paths, thereby improving the accuracy of the obtained second feature and ultimately enhancing the matching degree and accuracy of personalized proactive services provided in the intelligent cockpit environment.

[0016] Interaction log data of target members on the target website reflects their page interests and / or browsing habits during browsing, revealing their personalized interests. Therefore, by using this interaction log data as interaction data, the target member's state characteristics can be obtained. This, combined with their personalized interests, improves the accuracy of the obtained state characteristics, thus enhancing the accuracy of the obtained member state and improving the matching degree and accuracy of personalized proactive services in the intelligent cockpit environment. When interaction data includes interaction log data, user behavior sequences are generated based on the interaction log data, and these sequences are then sequence-encoded to obtain a first feature vector. This allows for the full utilization of the target member's serialized behavior information on the target website, improving the understanding of their interaction behavior and thus improving the accuracy of the obtained first feature vector. This, in turn, improves the accuracy of the obtained second feature vector, further enhancing the matching degree and accuracy of personalized proactive services in the intelligent cockpit environment.

[0017] The interaction feedback data of the target member in the target mini-program reflects the data related to the target member's personalized feedback during the use of the mini-program. Therefore, by using the interaction feedback data of the target member in the target mini-program as interaction data, the target member's state characteristics can be obtained based on this data. On this basis, it can help to combine the target member's personalized interaction feedback information in the mini-program to obtain state characteristics, thereby improving the accuracy of the obtained state characteristics of the target member, and thus helping to improve the matching degree and accuracy of personalized proactive services provided in the smart cockpit environment. When the interaction data includes interaction feedback data, the second feature vector corresponding to the interaction feedback data is obtained by feature encoding of the interaction feedback data, which can be conveniently obtained.

[0018] The page navigation path of a target member on the target client can reflect their personalized client usage habits. Therefore, by using the page navigation path as interaction data, the target member's state characteristics can be obtained based on these personalized habits. This improves the accuracy of the obtained state characteristics, thereby enhancing the accuracy of member status and the matching degree and accuracy of personalized proactive services in the smart cockpit environment. When the interaction data includes page navigation paths, the path is processed into a directed graph, and then encoded using a graph neural network to obtain the corresponding third feature vector. This captures the navigation relationships between different business units, reflecting the complex behavioral intentions of the target member. This improves the accuracy of the obtained third feature vector, further enhancing the accuracy of the obtained second feature vector and ultimately improving the matching degree and accuracy of personalized proactive services in the smart cockpit environment.

[0019] In some optional embodiments, the interaction log data includes at least one of the following: vehicle model browsing information, page dwell time, vehicle configuration selection information, and vehicle model comparison behavior information.

[0020] Accordingly, the acquired interaction log data can include various types of interaction data, such as vehicle model browsing information, page dwell time, vehicle configuration selection information, and vehicle model comparison behavior information. Since vehicle model browsing information reflects the vehicle models that the target member is interested in, page dwell time reflects the target member's level of interest in the information content on the page (such as vehicle models), vehicle configuration selection information reflects the vehicle configuration types that the target member is interested in, and vehicle model comparison behavior information reflects the differences between different vehicle models that the target member is interested in, the various types of interaction data can comprehensively reflect the personalized interest characteristics of the target member on the target website. Based on this, it helps to further improve the accuracy of the obtained target member status characteristics, thereby improving the accuracy of the obtained member status, and also helps to further improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0021] In some optional embodiments, the interactive feedback data includes at least one of text data and behavioral data, wherein the text data includes at least one of comment text and voice message composing text, and the behavioral data includes at least one of rating data, facial expression data, and dwell time before operating a preset control.

[0022] Accordingly, the obtained interactive feedback data can include text data such as comment text and voice message composing text, as well as behavioral data such as rating data, facial expression data, and dwell time before operating preset controls. That is, it includes both explicit feedback data of the target member when interacting, such as rating data and facial expression data, and implicit feedback data of the target member when interacting, such as voice message composing text and dwell time before operating preset controls. By combining feedback data from different dimensions, a second feature vector of the target member can be obtained, which helps to improve the accuracy of the obtained second feature vector. Based on this, it helps to further improve the accuracy of the obtained state characteristics of the target member, and thus helps to improve the accuracy of the obtained member state. It also helps to further improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0023] In some optional embodiments, the directed graph is a directed weighted graph, and the weight of the edge in the directed weighted graph is determined based on the page interaction metrics between the two business units connected by the edge.

[0024] Accordingly, the directed graph obtained based on the page navigation path is a directed weighted graph. The weight of the edge in the directed weighted graph is determined based on the page interaction indicators between the two business units connected by the edge. This weight can be combined with the page interaction indicators between the user object and the two business units to comprehensively determine the third feature vector, which helps to improve the accuracy of the obtained third feature vector. On this basis, it helps to further improve the accuracy of the obtained target member's state characteristics, thereby improving the accuracy of the obtained member state. It also helps to further improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0025] In some optional embodiments, page interaction metrics include at least one of the following: the number of jumps between two business units connected by an edge, and the average dwell time of the target member in the two business units.

[0026] Accordingly, the page interaction metrics used to determine the weights of edges in a directed weighted graph include at least one of the following: the number of jumps between two business units connected by an edge, and the average dwell time of a target member between the two business units connected by an edge. Since the number of jumps between two business units connected by an edge and the average dwell time between the two business units reflect the target member's interest in the business units and the relationship between them, the third feature vector can be comprehensively determined by combining these with the page interaction metrics of the target member between the two business units. This helps to improve the accuracy of the obtained third feature vector, and on this basis, it helps to further improve the accuracy of the obtained state characteristics of the target member, thereby improving the accuracy of the obtained member state, and also helps to further improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0027] In some optional embodiments, the user behavior sequence is sequence encoded to obtain a first feature vector, including:

[0028] Map each behavioral event in the user behavior sequence to an embedding vector;

[0029] Based on the sequence order of user behavior sequences, the embedding vectors of each behavior event are encoded to obtain an encoded sequence;

[0030] The encoded sequence is subjected to self-attention pooling to obtain a first feature vector of a predetermined length.

[0031] Accordingly, in the process of sequence encoding of user behavior sequences to obtain the first feature vector, each behavior event in the user behavior sequence is first mapped to an embedding vector. Then, the embedding vectors of each behavior event are encoded in combination with the sequence data of the user behavior sequence to obtain an encoded sequence. On this basis, the encoded sequence is subjected to self-attention pooling to obtain the first feature vector of a predetermined length. This can improve the efficiency of obtaining the first feature vector while capturing the global information of the user object.

[0032] In some optional embodiments, feature encoding is performed on the interactive feedback data to obtain a second feature vector, including:

[0033] Extract features from the text data in the interactive feedback data to obtain semantic feature vectors;

[0034] Feature extraction is performed on behavioral data in interactive feedback data to obtain behavioral feature vectors;

[0035] By combining the semantic feature vector and the behavioral feature vector, a second feature vector is obtained.

[0036] Accordingly, in the process of feature encoding the interactive feedback data to obtain the second feature vector, different methods are used to extract features from the text data and behavioral data in the interactive feedback data to obtain semantic feature vectors and behavioral feature vectors respectively. The obtained semantic feature vectors and behavioral feature vectors are then combined to obtain the second feature vector corresponding to the interactive feedback data. This allows for feature extraction using corresponding methods according to different types of interactive feedback data, making the extracted features more consistent with the characteristics of the corresponding type of interactive feedback data. This helps to improve the accuracy of the obtained second feature vector. Based on this, it helps to further improve the accuracy of the obtained target member's state features, thereby improving the accuracy of the obtained member state and also helping to improve the matching degree and accuracy of personalized proactive services provided in the intelligent cockpit environment.

[0037] In some alternative embodiments, a graph neural network is used to encode the directed graph to obtain a third feature vector, including:

[0038] Based on the data information of the business units corresponding to the nodes in the directed graph, the nodes are encoded to obtain the node characteristics of each node. The data information includes at least one of the following: business unit type, business unit dwell time, and business unit bounce rate.

[0039] The node features of each node are processed by a graph attention layer to obtain an updated directed graph.

[0040] The updated directed graph is subjected to global pooling to obtain the third feature vector.

[0041] Accordingly, in the process of encoding a directed graph to obtain the third feature vector, the nodes are initially encoded based on data such as the business unit type, business unit dwell time, and business unit bounce rate of each node in the directed graph. After obtaining the node features of each node, the node features of each node are processed by a graph attention layer to obtain an updated directed graph. Then, the updated directed graph is processed by global pooling to obtain the third feature vector. This helps to improve the accuracy of the obtained third feature vector. Based on this, it helps to further improve the accuracy of the obtained target member state features, thereby improving the accuracy of the obtained member state. It also helps to improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0042] In some optional embodiments, obtaining in-vehicle perception data corresponding to the target occupants inside the vehicle includes:

[0043] Acquire at least one type of in-vehicle interaction data generated by the interaction between the target member and the vehicle;

[0044] Acquire physiological state data of target members collected through wearable devices;

[0045] The in-vehicle perception data corresponding to the target member is determined based on at least one of the in-vehicle interaction data and physiological state data.

[0046] Accordingly, in the process of acquiring in-vehicle perception data corresponding to the target occupant, the data is determined based on at least one of the in-vehicle interaction data generated by the interaction between the target occupant and the vehicle, and the physiological state data of the target occupant collected through wearable devices. Since the target occupant interacts with the vehicle based on their potential personalized behaviors, by using at least one in-vehicle interaction data generated by the interaction between the target occupant and the vehicle as in-vehicle perception data, personalized interaction information generated by the interaction between the target occupant and the vehicle can be obtained. This helps to improve the accuracy of the obtained state characteristics of the target occupant, and thus helps to improve the accuracy of the obtained occupant state. Physiological state data reflects the physiological state of the target occupant. Therefore, by using the physiological state data of the target occupant as in-vehicle perception data, the state characteristics of the target occupant can be obtained by combining the physiological state data of the target occupant. On this basis, when determining the occupant state based on the state characteristics of the target occupant and providing personalized proactive services accordingly, it helps to combine the physiological state of the target occupant to provide personalized proactive services, which helps to improve the accuracy of the obtained occupant state and also helps to improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0047] In some optional embodiments, the in-vehicle interaction data includes at least one of touch data, voice data, and physical operation data; wherein, the touch data includes at least one of touch coordinates, touch pressure value, touch area, and touch click interval; the voice data includes at least one of voice command text, acoustic features, and tone energy; and the physical operation data includes at least one of the rotation angle of physical knobs, the pressing depth of physical buttons, and the change in steering wheel grip force.

[0048] Accordingly, when acquiring in-vehicle interaction data, different types of data can be obtained, such as touch data, voice data, and physical operation data. Touch data, in addition to touch coordinates, can also include touch pressure values, touch area, and touch click intervals. Voice data, in addition to voice command text, can also include acoustic features and tone energy. Physical operation data can simultaneously include the rotation angle of physical knobs, the pressing depth of physical buttons, and changes in steering wheel grip force. The information such as touch pressure values, touch area, touch click intervals, acoustic features, tone energy, the rotation angle of physical knobs, the pressing depth of physical buttons, and changes in steering wheel grip force are closely related to the personalized characteristics of the target occupant. Therefore, the accuracy of the target occupant's state characteristics can be further obtained based on this information. On this basis, when determining the occupant's state based on the target occupant's state characteristics and providing personalized proactive services accordingly, it helps to improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0049] In some alternative embodiments, the physiological state data include at least one of heart rate, heart rate variability, skin conductance response, and skin temperature.

[0050] Accordingly, the obtained physiological state data can include different physiological state data such as heart rate, heart rate variability, skin conductance response, and skin temperature. This allows the physiological state of the target member to be obtained from different dimensions. Based on this, when determining the member's state based on the target member's state characteristics and providing personalized proactive services accordingly, it can help to provide personalized proactive services by combining different types of physiological states of the target member. This will help improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0051] In some optional embodiments, feature extraction is performed on the in-vehicle perception data, including:

[0052] In response to in-vehicle perception data, including in-vehicle interaction data, based on the data type of in-vehicle interaction data, corresponding encoding methods are used to encode each data type to obtain the encoded sub-features corresponding to each data type.

[0053] Attention fusion processing is performed on the encoded sub-features corresponding to each data type to obtain the vehicle behavior feature vector, the first feature of which includes the vehicle behavior feature vector.

[0054] Therefore, in the process of extracting features from in-vehicle interaction data to obtain in-vehicle behavior feature vectors, the data type of the in-vehicle interaction data is considered. For each data type, a corresponding encoding method is used to encode it to obtain its corresponding coded sub-features. Then, attention fusion processing is performed on the coded sub-features corresponding to each data type to obtain the in-vehicle behavior feature vector. Thus, different data types of in-vehicle perception data can be encoded using different encoding methods, and a more suitable encoding method can be used. This is applicable to the data characteristics of different data types and helps to improve the accuracy of the obtained coded sub-features. On this basis, attention fusion processing is further performed to make full use of the complementary characteristics of the coded sub-features of different data types, which helps to improve the accuracy and generalization ability of the obtained in-vehicle behavior feature vectors.

[0055] In some optional embodiments, feature extraction from in-vehicle perception data includes:

[0056] In response to in-vehicle perception data, including physiological state data, the physiological state data is encoded using convolutional kernels of different scales to obtain multiple physiological features at different scales.

[0057] Multiple physiological features at different scales are spliced ​​together to obtain spliced ​​features;

[0058] Temporal attention pooling is applied to the concatenated features to obtain physiological state feature vectors, with the first feature including the physiological state feature vectors.

[0059] Accordingly, in the process of extracting features from physiological state data and obtaining physiological state feature vectors, the physiological state data is encoded by convolutional kernels of different scales to obtain multiple physiological features of different scales. These features are then concatenated to obtain concatenated features. Temporal attention pooling is then applied to the concatenated features to obtain the physiological state feature vector. By first encoding the physiological state data with convolutional kernels of different scales, both rapidly changing and slowly changing physiological features can be captured. The subsequent temporal attention pooling process, which fully utilizes the complementary characteristics of rapidly changing and slowly changing physiological features, helps to improve the accuracy and generalization ability of the obtained physiological state feature vector.

[0060] In some optional embodiments, the in-vehicle interaction data includes: touch data, voice data, and physical operation data; the coded sub-features include touch sub-features, voice sub-features, and operation sub-features;

[0061] Based on the data types of in-vehicle interaction data, corresponding encoding methods are used to encode each data type, obtaining the corresponding encoded sub-features, including:

[0062] The touch sequence is determined based on the touch data, and the touch sequence is sequence encoded to obtain touch sub-features;

[0063] Extract prosodic features from speech data, obtain the corresponding speech text, extract semantic features from the speech text, fuse semantic features and prosodic features to obtain speech sub-features;

[0064] Feature statistics are performed on the physical operation data to obtain at least one statistical feature for each physical operation, and the statistical features of each physical operation are fused to obtain operation sub-features.

[0065] Accordingly, when in-vehicle interaction data includes touch data, voice data, and physical operation data, for touch data, after determining the touch sequence based on the touch data, the touch sub-features are obtained by sequence encoding the touch sequence. For voice data, on the one hand, prosodic features of the voice data are extracted; on the other hand, after obtaining the corresponding voice text, semantic features of the voice text are extracted, and then the semantic features and prosodic features are fused to obtain the voice sub-features corresponding to the voice data. Thus, voice sub-features can be obtained by combining two different dimensions of voice prosody and voice content. For physical operation data, feature statistics are performed to obtain at least one statistical feature for each physical operation, and the statistical features are fused to obtain operation sub-features. Thus, operation sub-features can be obtained by combining the statistical characteristics of different dimensions of physical operations, which can improve the accuracy of the obtained touch sub-features, voice sub-features, and operation sub-features. This helps to improve the accuracy of the obtained target member's state features. On this basis, when determining the member's state based on the target member's state features and providing personalized proactive services accordingly, it can help improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0066] In some optional embodiments, the first feature includes more than one feature vector, and the second feature includes more than one feature vector;

[0067] The first and second features are fused to obtain the state features of the target member, including:

[0068] An input matrix is ​​constructed based on the first and second features, where each row of the input matrix represents a feature vector.

[0069] Based on the input matrix, calculate the attention weights between any two rows in the input matrix to obtain the attention weight matrix;

[0070] The input matrix and the attention weight matrix are multiplied together to obtain the state features of the target member.

[0071] Accordingly, in the process of fusing the first feature and the second feature to obtain the state features of the target member, the first feature and the second feature are constructed as an input matrix. Then, the attention weight matrix is ​​obtained by calculating the attention weight between any two rows in the input matrix. Finally, the attention weight matrix and the input matrix are multiplied to obtain the state features of the target member. That is, in the process of fusing the first feature and the second feature, the cross-modal attention fusion mechanism can be used to fuse the first feature and the second feature, which can make full use of the complementary characteristics of different features and help improve the accuracy of the obtained state features of the target member.

[0072] Secondly, this application also provides a vehicle occupant data processing apparatus, comprising:

[0073] The in-vehicle feature acquisition module is used to acquire in-vehicle perception data corresponding to the target occupants in the vehicle, and to extract features from the in-vehicle perception data to obtain the first feature;

[0074] The channel interaction feature acquisition module is used to acquire the interaction data of the target member in at least one interaction channel, and to extract features from the interaction data to obtain a second feature;

[0075] The feature fusion module is used to perform feature fusion processing on the first feature and the second feature to obtain the state features of the target member, and to determine the member state of the target member based on the state features.

[0076] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any of the above embodiments.

[0077] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods of any of the above embodiments.

[0078] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods of any of the above embodiments.

[0079] Regarding the beneficial effects of any of the technical solutions in the second to fifth aspects mentioned above, refer to the beneficial effects of the corresponding technical solutions in the first aspect; repeated examples will not be listed here. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0081] Figure 1 This is a schematic diagram of an optional application environment for a method of processing vehicle occupant data in one embodiment.

[0082] Figure 2 This is a schematic diagram of an optional process for processing vehicle occupant data in one embodiment;

[0083] Figure 3 This is a schematic diagram of an optional process for feature extraction from interactive data in one embodiment;

[0084] Figure 4 This is a schematic diagram of an optional process for obtaining a first feature vector by sequence encoding a user behavior sequence in one embodiment;

[0085] Figure 5 This is a schematic diagram of an optional process for feature encoding of interactive feedback data in one embodiment;

[0086] Figure 6 This is a schematic diagram of an optional process for encoding a directed graph using a graph neural network in one embodiment;

[0087] Figure 7 This is a schematic diagram of an optional process for obtaining in-vehicle perception data corresponding to target occupants in one embodiment;

[0088] Figure 8 This is a schematic diagram of an optional process for feature extraction of in-vehicle interaction data in one embodiment;

[0089] Figure 9 This is a schematic diagram of an optional process for encoding data types based on in-vehicle interaction data in one embodiment;

[0090] Figure 10 This is a schematic diagram of an optional process for extracting features from physiological state data in one embodiment;

[0091] Figure 11 This is a schematic diagram of an optional process for obtaining state features based on a first feature and a second feature in one embodiment;

[0092] Figure 12This is a schematic diagram of an optional overall architecture for a method of processing vehicle occupant data in one embodiment.

[0093] Figure 13 This is a schematic diagram of an optional architecture of the encoder included in a multimodal feature encoding module in one embodiment;

[0094] Figure 14 This is a schematic diagram of an optional deployment architecture for a method of processing vehicle occupant data in one embodiment;

[0095] Figure 15 This is a schematic diagram of an optional principle architecture for obtaining the member status of a target member in one embodiment;

[0096] Figure 16 This is a schematic diagram of an optional structure of a vehicle occupant data processing device in one embodiment. Detailed Implementation

[0097] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0098] The terms "first," "second," etc., used in this application may be used to describe various elements, but these elements are not limited by these terms. These terms are used only to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0099] Currently, in technologies related to processing vehicle occupant data in intelligent cockpit environments to provide personalized proactive services based on the processing results, in order to achieve accurate perception of the status of vehicle occupants (drivers or passengers) and obtain accurate processing results of vehicle occupant data, either a perception scheme based on a single data source is adopted, such as collecting driver facial video through an in-vehicle camera and using computer vision technology to identify behaviors such as blinking frequency and yawning to determine fatigue status; or collecting heart rate variability through wearable wristbands and using time series models to analyze the user's stress level, etc., or a simple fusion processing based on data from multiple sensors is adopted, such as simply concatenating the feature vectors extracted from different sensors in terms of dimensions to form a new long vector, which is then input into a classifier (such as a support vector machine) to determine whether the driver is in a distracted state, etc. However, the data sources for processing vehicle occupant data in these related technologies are limited to in-vehicle data, such as focusing only on in-vehicle sensor data (e.g., camera, steering wheel) or single wearable device data. The data sources are limited, and even when multiple data sources are used, the final state features are often obtained by feature stitching. With limited data sources, the state of the vehicle occupants obtained accordingly is also limited, making it difficult to obtain data processing results that accurately and comprehensively reflect the personalized state of the vehicle occupants.

[0100] Research has revealed that users outside the vehicle generate a wealth of personalized data containing rich intentions and preferences through digital touchpoints. For example, interactive data generated before or during a trip through websites (such as car manufacturer websites), clients, and lightweight applications (such as mini-programs) can be combined with in-vehicle perception data. This allows for the processing of rich personalized data about vehicle occupants to obtain their status characteristics. The resulting occupant status can more comprehensively reflect their individual personalities. Based on this, personalized proactive services can be provided in the smart cockpit environment, which also helps to improve the adaptability of the personalized proactive services provided.

[0101] Accordingly, the vehicle occupant data processing method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, one or more sensors 101 may be installed inside the vehicle 10. The processor 102 is connected to one or more sensors 101. The processor 102 can obtain in-vehicle perception data of vehicle occupants through one or more sensors 101. The first server 20 can provide one or more interaction channels. Vehicle occupants can request backend services provided by the first server 20 through terminals and different interaction channels (such as websites, clients, or mini-programs). The first server 20 can obtain the interaction data of vehicle occupants on one or more interaction channels.

[0102] In some feasible embodiments, vehicle 10 and first server 20 are connected via network communication. First server 20 can acquire in-vehicle perception data provided by vehicle 10, and combine this in-vehicle perception data with interaction data from at least one interaction channel to obtain the state characteristics of the vehicle member (i.e., the target member), thereby determining the member's status. First server 20 can determine personalized service instructions for the target member based on the obtained member status. In related examples, the member status obtained by first server 20 can also be provided to vehicle 10, which then determines personalized service instructions for the target member based on the obtained member status. In some examples, vehicle 10 can also obtain the target member's interaction data from at least one interaction channel from first server 20, and combine this with the target member's in-vehicle perception data and interaction data to obtain the target member's status, and then determine personalized service instructions for the target member based on the obtained member status. In related examples, the member status obtained by vehicle 10 can also be provided to first server 20, which then determines personalized service instructions for the target member based on the obtained member status.

[0103] In some feasible embodiments, the application environment may also involve a second server 30, which can be connected to the vehicle 10 and the first server 20 via network communication. The second server 30 can obtain in-vehicle perception data provided by the vehicle 10 and interaction data of the target member in at least one interaction channel provided by the first server 20. Combining the in-vehicle perception data and the interaction data, the second server 30 obtains the member status of the target member and provides the obtained member status to the vehicle 10 or the first server 20. The vehicle 10 or the first server 20 then determines personalized service instructions for the target member based on the obtained member status.

[0104] The data storage system can store the data that the first server 20 and / or the second server 30 need to process. The data storage system can be integrated onto the first server 20 and / or the second server 30, or it can be located in the cloud or on other network servers. The first server 20 and the second server 30 can be implemented using independent servers or a server cluster consisting of multiple servers.

[0105] In one exemplary embodiment, such as Figure 2 As shown, a method for processing vehicle occupant data is provided, which can be applied to... Figure 1 Taking vehicle 10 or the first server 20 as an example, the explanation includes the following steps 201 to 203. Wherein:

[0106] Step 201: Obtain in-vehicle perception data corresponding to the target occupants inside the vehicle, and extract features from the in-vehicle perception data to obtain the first feature.

[0107] The target occupant inside the vehicle is the object whose state characteristics need to be determined, which can be based on the identified vehicle occupants. In some feasible embodiments, if only one occupant is identified, that occupant can be directly designated as the target occupant. In other feasible embodiments, if two or more occupants are identified, the target occupant can be determined from among the two or more occupants. In related examples, one occupant can be selected as the target occupant from among the two or more occupants, or each occupant can be designated as a separate target occupant, i.e., for each identified vehicle occupant, a separate target occupant can be designated to determine the occupant state of each vehicle occupant. Based on this, personalized proactive services can be provided to each vehicle occupant according to their respective occupant state. The method of identifying vehicle occupants and determining the target occupant is not specifically limited in the embodiments of this application, as long as vehicle occupants can be identified and target occupants can be determined.

[0108] In-vehicle perception data refers to the state data of a target occupant obtained by perceiving the target occupant's state while they are inside the vehicle or within a certain range of the vehicle. By extracting features from the obtained in-vehicle perception data, corresponding features can be obtained, referred to as the first feature in this embodiment.

[0109] Step 202: Obtain the interaction data of the target member in at least one interaction channel, and extract features from the interaction data to obtain the second feature.

[0110] An interaction channel refers to the medium or path through which users and products or services exchange information. Interaction data refers to the data generated during the two-way information exchange and behavioral interaction between target members and products or services through the interaction channel. Based on the obtained interaction data, corresponding features can be obtained by extracting features from the interaction data; in this embodiment, these are referred to as the second feature.

[0111] Step 203: Perform feature fusion processing on the first feature and the second feature to obtain the state features of the target member, and determine the member state of the target member based on the state features.

[0112] For the first and second features of the target member, feature fusion processing can be performed, and the state features obtained after feature fusion processing can be used as the global feature vector of the target member. Based on the state features, the member state of the target member can be determined.

[0113] The vehicle occupant data processing method based on the embodiments described above involves obtaining in-vehicle perception data corresponding to a target occupant, extracting features from the in-vehicle perception data to obtain a first feature, and simultaneously acquiring interaction data of the target occupant on at least one interaction channel, extracting features from the interaction data to obtain a second feature, and then performing feature fusion processing on the first and second features to obtain the target occupant's state features. Based on these state features, the occupant's occupant state is determined. In other words, during the processing of vehicle occupant data, the global state features obtained from processing the vehicle occupant data consider both the target occupant's state within the vehicle (i.e., the in-vehicle perception data corresponding to the target occupant) and the target occupant's interaction data on at least one interaction channel. When target members interact through interactive channels, personalized channel interactions are typically achieved. This means that the interaction data of target members through interactive channels can reflect the user's personalized behavioral characteristics. Based on this, the first feature obtained from in-vehicle perception data and the second feature obtained from interaction data are fused to obtain state features. This state feature can reflect both the personalized state characteristics of the target member when in the vehicle and when interacting through interactive channels. Using this state feature as a global feature vector to determine the member's state can improve the accuracy of obtaining the target member's state. Based on this, personalized proactive services can be provided in the intelligent cockpit environment, which can also help improve the accuracy of the personalized proactive services provided.

[0114] The specific type of interaction channel is not limited. In some feasible embodiments, the interaction channel may include at least one of a target website, a target mini-program, and a target client. In this case, the interaction data includes at least one of the following: interaction log data obtained from the target website; interaction feedback data obtained from the target mini-program; and page jump path obtained from the target client.

[0115] The target website refers to the website for which interaction log data needs to be obtained. It can be a website that provides interactive services to users. In the relevant embodiments of this application, the target website can be a website that provides vehicle services to users (including but not limited to vehicle model viewing, vehicle configuration information viewing, vehicle price comparison, etc.). Some examples of target websites may include the official website of car manufacturers, but are not limited to this.

[0116] Mini-programs refer to lightweight applications that need to be attached to a host application, do not require installation, and can be used and then discarded. Target mini-programs refer to mini-programs that need to obtain interactive feedback data through interaction with the target mini-program. In the relevant embodiments of this application, the target mini-program may be a mini-program that provides vehicle services to users (including but not limited to vehicle model viewing, vehicle configuration information viewing, vehicle price comparison, vehicle after-sales service, etc.). In some examples, the target mini-program may include mini-programs provided by car manufacturers, but it is not limited to this.

[0117] A client refers to an application that can be installed and run independently on a terminal. A target client refers to a client that needs to obtain data such as page navigation paths for interaction. In the relevant embodiments of this application, the target client may be a client that provides vehicle services to users (including but not limited to vehicle model viewing, vehicle configuration information viewing, vehicle price comparison, vehicle after-sales service, etc.). In some examples, the target client may include clients provided by car manufacturers, but it is not limited to these.

[0118] Accordingly, the interaction data of the target member on at least one interaction channel may include at least one of the following: interaction log data obtained from the target website, interaction feedback data obtained from the target mini-program, and page jump paths obtained from the target client, wherein:

[0119] The interaction log data of target members on the target website reflects the page objects they are interested in and / or their browsing habits during the browsing process. It can reflect the personalized interest characteristics of the target members. Therefore, by using the interaction log data of target members on the target website as interaction data, it is possible to obtain the status characteristics of the target members based on the interaction log data. On this basis, it can help to combine the personalized interest characteristics of the target members on the target website to obtain status characteristics, thereby improving the accuracy of the obtained status characteristics of the target members. This also helps to improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0120] The interaction feedback data of the target member in the target mini-program reflects the data related to the target member's personalized feedback during the use of the target mini-program. Therefore, by using the interaction feedback data of the target member in the target mini-program as interaction data, it is possible to obtain the target member's status characteristics based on the interaction feedback data of the target member in the target mini-program. On this basis, it can help to combine the target member's personalized interaction feedback information in the target mini-program to obtain status characteristics, thereby improving the accuracy of obtaining the target member's status characteristics, and thus helping to improve the matching degree and accuracy of providing personalized proactive services in the smart cockpit environment.

[0121] The page navigation path of a target member on the target client can reflect the target member's personalized usage habits during the use of the target client. Therefore, by using the page navigation path of a target member on the target client as interaction data, the target member's status characteristics can be obtained based on the personalized habits reflected in the page navigation path. On this basis, the accuracy of the obtained status characteristics of the target member can be improved, which in turn helps to improve the accuracy of the obtained member status and also helps to improve the matching degree and accuracy of providing personalized proactive services in the smart cockpit environment.

[0122] The specific data type of the interaction log data obtained from the target website is not limited. In some optional embodiments, the interaction log data includes at least one of the following: vehicle model browsing information, page dwell time, vehicle configuration selection information, and vehicle model comparison behavior information.

[0123] Vehicle model browsing information refers to information about vehicle models or vehicle-related pages that a target member browses during their visit to the target website, or during their visit to the target website within a predetermined time frame. The method of obtaining vehicle model browsing information is not limited. In relevant examples, it can be determined based on vehicle model information contained in pages visited by the target member, vehicle model information associated with page addresses, or page type, but is not limited to these methods.

[0124] Page dwell time refers to the duration of time a target member stays on a page. In relevant examples, it can be determined based on the time difference between the start time when the target member opens the page (such as the access time or request time) and the departure time when the target member leaves the page (such as closing the page or switching to requesting to open another page), but it is not limited to this.

[0125] Vehicle configuration selection information refers to the vehicle configuration information that a target member selects or views during their visit to the target website, including but not limited to color information, wheel information, etc. There are no restrictions on how vehicle configuration selection information is obtained.

[0126] Vehicle comparison behavior information refers to whether a target member compares two or more vehicle models while visiting a target website, and if so, the specific information related to the vehicle comparison. The method of obtaining vehicle comparison behavior information is not specifically limited in this application embodiment.

[0127] Accordingly, the acquired interaction log data can include various types of interaction data, such as vehicle model browsing information, page dwell time, vehicle configuration selection information, and vehicle model comparison behavior information. Since vehicle model browsing information reflects the vehicle models that the target member is interested in, page dwell time reflects the target member's level of interest in the information content on the page (such as vehicle models), vehicle configuration selection information reflects the vehicle configuration types that the target member is interested in, and vehicle model comparison behavior information reflects the differences between different vehicle models that the target member is interested in, the various types of interaction data can comprehensively reflect the personalized interest characteristics of the target member on the target website. Based on this, it helps to further improve the accuracy of the obtained target member status characteristics, thereby improving the accuracy of the obtained member status, and also helps to further improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0128] When the interaction data includes interaction log data, refer to Figure 3 As shown, in some optional embodiments, feature extraction is performed on the interaction data to obtain a second feature, including:

[0129] Step 301: In response to the interaction data, including the interaction log data, generate a user behavior sequence based on the interaction log data, and perform sequence encoding on the user behavior sequence to obtain a first feature vector.

[0130] Step 302: In response to the interaction data, including the interaction feedback data, perform feature encoding on the interaction feedback data to obtain a second feature vector.

[0131] Step 303: In response to the interaction data including the page jump path, the page jump path is preprocessed into a directed graph, where the nodes in the directed graph represent business units in the target client, and the edges in the directed graph represent the jump relationship between two connected business units; the directed graph is encoded using a graph neural network to obtain the third feature vector.

[0132] Step 304: Determine the second feature based on at least one of the first feature vector, the second feature vector, and the third feature vector.

[0133] Accordingly, the second feature is determined based on at least one of the first feature vector corresponding to the interaction log data, the second feature vector corresponding to the interaction feedback data, and the third feature vector corresponding to the page jump path. This allows for a full integration of the personalized characteristics reflected by the interaction log data, interaction feedback data, and page jump path, thereby improving the accuracy of the obtained second feature. This, in turn, helps to improve the matching degree and accuracy of personalized proactive services provided in the intelligent cockpit environment.

[0134] When the interaction data includes interaction log data, the user behavior sequence is generated based on the interaction log data, and the user behavior sequence is sequence encoded to obtain the first feature vector based on the interaction log data. This enables full use of the serialized behavior information of the target members on the target website, improves the understanding of the target members' interaction behavior on the target website, helps to improve the accuracy of the obtained first feature vector, and further helps to improve the accuracy of the obtained second feature. On this basis, it improves the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0135] When the interaction data includes interaction feedback data, the second feature vector corresponding to the interaction feedback data is obtained by feature encoding the interaction feedback data, thereby making it easy to obtain the second feature vector corresponding to the interaction feedback data.

[0136] When the interaction data includes page navigation paths, the process involves processing the page navigation paths into a directed graph and then encoding the directed graph using a graph neural network to obtain the third feature vector corresponding to the page navigation path. This vector can capture the navigation relationships between different business units and reflect the complex behavioral intentions of the target members, thereby improving the accuracy of the obtained third feature vector. This, in turn, helps to improve the accuracy of the obtained second feature vector, and on this basis, improves the matching degree and accuracy of personalized proactive services provided in the intelligent cockpit environment.

[0137] In step 301, the method for generating user behavior sequences based on interaction log data is not limited. In relevant examples, the interaction log data can be segmented on a session-by-session basis to obtain user behavior sequences. In some feasible embodiments, based on the obtained interaction log data, the interaction log data can be preprocessed by performing log cleaning (e.g., removing invalid data, noisy data, and supplementing necessary fields), and then the preprocessed interaction log data can be segmented to obtain user behavior sequences. The specific methods of preprocessing the interaction log data, such as log cleaning, are not specifically limited in this application embodiment.

[0138] Based on the obtained user behavior sequence, sequence encoding can be performed on the user behavior sequence to obtain a first feature vector. The sequence encoding of the user behavior sequence can be performed in various possible ways; in some optional embodiments, refer to... Figure 4 As shown, the user behavior sequence is sequence encoded to obtain the first feature vector, including:

[0139] Step 401: Map each behavioral event in the user behavior sequence to an embedding vector.

[0140] Mapping behavioral events to embedding vectors means representing the behavioral event using a continuous vector of predetermined length. There are no restrictions on the methods used to map behavioral events to embedding vectors; the following examples illustrate only a few types of behavioral events:

[0141] Taking the event type of a behavior event as an example, it can be represented as an event type vector based on the event type. Some possible event types may include: entering the page (page_view), click event (click), scroll event (scroll), comparison event (comparison), etc., but are not limited to these.

[0142] Taking the page type corresponding to the behavior event as an example, different page types can be represented as corresponding page type vectors. Some examples of page types may include: homepage, vehicle model page, configuration page, comparison page, finance page, appointment page, community page, help page, etc., but are not limited to these.

[0143] Taking the dwell time corresponding to the behavior event as an example, multiple dwell time periods can be discretized (for example, some examples can be divided into 5 dwell time periods: less than 5s, 5s-15s, 15s-30s, 30s-60s, and greater than 60s). The corresponding embedding vector is determined for the dwell time period in which the page dwell time is located.

[0144] Taking the configuration options corresponding to behavior events as an example, the corresponding vector can be set independently for each configuration option.

[0145] It is understood that in other embodiments, other methods may be used to obtain the embedding vectors corresponding to each behavioral event, and this application embodiment does not impose specific limitations on this.

[0146] Step 402: Based on the sequence order of the user behavior sequence, encode the embedding vector of each behavior event to obtain the encoded sequence.

[0147] For the sequence data of the obtained user behavior sequence, the embedding vector of each behavior event can be encoded to obtain the encoded vector. The specific encoding method can be various possible methods, and no specific limitation is made in this embodiment.

[0148] Step 403: Perform self-attention pooling on the encoded sequence to obtain a first feature vector of a predetermined length.

[0149] The obtained encoded sequence can be processed by self-attention pooling to capture the temporal dependencies between different behavioral events, thereby obtaining a first feature vector of a predetermined length.

[0150] In relevant examples, a sequence encoder can be used to encode user behavior sequences to obtain a first feature vector of a predetermined length. In some examples, the sequence encoder can be a recurrent neural network-based sequence encoder or a Transformer-based sequence encoder. In other examples, the sequence encoder can be an attention-based behavior sequence encoder. Taking an attention-based behavior sequence encoder as an example, the behavior sequence encoder can include an input layer, an encoding layer, and an output layer, wherein:

[0151] The input layer maps each behavioral event in the user behavior sequence to an embedding vector, for example, a 32-dimensional embedding vector;

[0152] In relevant examples, the hidden layer can be a two-layer GRU (Gated Recurrent Unit) structure. The hidden layer encodes the embedding vector of each behavioral event to obtain an encoded sequence.

[0153] The output layer performs self-attention pooling on the encoded sequence, aggregating the variable-length encoded sequence into a first feature vector of a predetermined length, which is a vector rich in temporal information.

[0154] Accordingly, in the process of sequence encoding of user behavior sequences to obtain the first feature vector, each behavior event in the user behavior sequence is first mapped to an embedding vector. Then, the embedding vectors of each behavior event are encoded in combination with the sequence data of the user behavior sequence to obtain an encoded sequence. On this basis, the encoded sequence is subjected to self-attention pooling to obtain the first feature vector of a predetermined length. This can improve the efficiency of obtaining the first feature vector while capturing the global information of the user object.

[0155] The specific data type of the interactive feedback data obtained from the target mini-program is not limited. In some optional embodiments, the interactive feedback data includes at least one of text data and behavioral data. The text data includes at least one of comment text and voice message writing text. The behavioral data includes at least one of rating data, emoticon data, and dwell time before operating a preset control.

[0156] Comment text refers to the text data entered by target members when commenting on relevant pages or information during interaction with the target mini-program.

[0157] Voice message transcription refers to the text data obtained by converting the voice data input by the target member into text during the interaction process through the target mini-program.

[0158] Rating data refers to the rating data, such as star ratings, that users actively provide on relevant rating pages during interactions with the target mini-program. Rating data can quantitatively reflect the subjective preferences and satisfaction levels of target users during the interaction process.

[0159] Facial expression data refers to the emotional expressions selected by users during interactions with the target mini-program. Facial expression data can reflect the emotional or attitudinal tendencies actively expressed by users during the interaction.

[0160] The dwell time before operating the preset control refers to the length of time the target member stays on the preset control (such as a button, input box, tab, slider, etc.). This dwell time can reflect the target member's intention to operate the preset control, such as the degree of interest, whether it is a mistake, etc., but is not limited to this.

[0161] Accordingly, the obtained interactive feedback data can include text data such as comment text and voice message composing text, as well as behavioral data such as rating data, facial expression data, and dwell time before operating preset controls. That is, it includes both explicit feedback data of the target member when interacting, such as rating data and facial expression data, and implicit feedback data of the target member when interacting, such as voice message composing text and dwell time before operating preset controls. By combining feedback data from different dimensions, a second feature vector of the target member can be obtained, which helps to improve the accuracy of the obtained second feature vector. Based on this, it helps to further improve the accuracy of the obtained state characteristics of the target member, and thus helps to improve the accuracy of the obtained member state. It also helps to further improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0162] The method of feature encoding the interactive feedback data to obtain the second feature vector is not limited; in some optional embodiments, refer to... Figure 5 As shown, feature encoding is performed on the interactive feedback data to obtain a second feature vector, including:

[0163] Step 501: Extract features from the text data in the interactive feedback data to obtain semantic feature vectors.

[0164] The text data can include comment text and voice message writing text as described above. Semantic features can be extracted from the comment text and voice message writing text respectively to obtain the semantic feature vectors corresponding to the comment text and voice message writing text.

[0165] There are no restrictions on the methods for extracting features from text data. In some examples, a pre-trained, defined language model (such as a simplified version of BERT (Bidirectional Encoder Representations from Transformers, a deep bidirectional pre-trained language model based on Transformer Encoder)) can be used to extract semantic features and obtain semantic features.

[0166] Step 502: Extract features from the behavioral data in the interactive feedback data to obtain behavioral feature vectors.

[0167] The behavioral data can include, as mentioned above, rating data, facial expression data, and dwell time before operating preset controls. Feature extraction can be performed on the rating data, facial expression data, and dwell time before operating preset controls to obtain the behavioral features corresponding to each of the rating data, facial expression data, and dwell time before operating preset controls.

[0168] There are no restrictions on the methods for feature extraction from behavioral data. In some examples, multilayer perceptrons can be used to extract behavioral features from behavioral data.

[0169] Step 503: Combine the semantic feature vector and the behavioral feature vector to obtain the second feature vector.

[0170] Based on the obtained semantic feature vector and behavioral feature vector, the semantic feature vector and behavioral feature vector can be combined to obtain a second feature vector.

[0171] In some feasible examples, a lightweight multimodal encoder can be used to encode features from the interactive feedback data to obtain a second feature vector. This lightweight multimodal encoder may include a text encoding branch, a behavior encoding branch, and a fusion layer, wherein:

[0172] The text encoding branch is used to extract features from the text data in the interactive feedback data to obtain semantic feature vectors. In relevant examples, the DistilBERT-base model (a knowledge distillation-based model that transfers knowledge from a BERT-base teacher model to a student model with reduced parameters) can be used to extract semantic features from the text data to obtain semantic feature vectors. In some feasible embodiments, after obtaining the semantic features corresponding to the comment text and the semantic feature vectors corresponding to the voice message writing text, the text encoding branch can perform fusion processing such as concatenation on the semantic feature vectors corresponding to the comment text and the semantic feature vectors corresponding to the voice message writing text before outputting them. Alternatively, the comment text and the voice message writing text can be combined, and semantic features can be extracted from the combined text before outputting it. In other feasible embodiments, other methods can also be used to output semantic features.

[0173] The behavior encoding branch is used to extract features from the behavior data in the interaction feedback data to obtain behavior feature vectors. In relevant examples, the embedding vectors corresponding to the rating data, expression data, and dwell time before the operation of the preset control can be obtained by a multilayer perceptron. The embedding vectors corresponding to the rating data, expression data, and dwell time before the operation of the preset control are then concatenated to obtain the concatenated behavior feature vector for output.

[0174] The fusion layer is used to fuse the semantic feature vector output by the text encoding branch and the behavioral feature vector output by the behavioral encoding branch to obtain a second feature vector. The specific fusion method is not limited. In some examples, the semantic feature vector and the behavioral feature vector can be linearly weighted and then non-linearly compressed to obtain the fused second feature vector.

[0175] In some specific examples, the features obtained by linearly weighting and summing the semantic feature vector and the behavioral feature vector can be expressed by the following formula: .

[0176] The second eigenvector obtained after nonlinear compression can be expressed by the formula:

[0177] .

[0178] in, The features are obtained by performing a linear weighted summation of semantic feature vectors and behavioral feature vectors. For semantic feature vectors, semantic feature vector The weighted matrix, For behavioral feature vectors, behavioral feature vector The weighted matrix, This is the second feature vector obtained after fusion. It is a non-linear activation function. is a projection matrix with non-linear activation.

[0179] In some feasible examples, the obtained second feature vector can be mapped to a preset emotion space, such as a three-dimensional emotion space (valence (reflecting the positive or negative degree of emotion), arousal (reflecting the excitement or calmness of emotion), dominance (reflecting the degree of dominance or influence of emotion)), but is not limited to this.

[0180] Accordingly, in the process of feature encoding the interactive feedback data to obtain the second feature vector, different methods are used to extract features from the text data and behavioral data in the interactive feedback data to obtain semantic feature vectors and behavioral feature vectors respectively. The obtained semantic feature vectors and behavioral feature vectors are then combined to obtain the second feature vector corresponding to the interactive feedback data. This allows for feature extraction using corresponding methods according to different types of interactive feedback data, making the extracted features more consistent with the characteristics of the corresponding type of interactive feedback data. This helps to improve the accuracy of the obtained second feature vector. Based on this, it helps to further improve the accuracy of the obtained target member's state features, thereby improving the accuracy of the obtained member state and also helping to improve the matching degree and accuracy of personalized proactive services provided in the intelligent cockpit environment.

[0181] The page navigation path obtained for the target member on the target client refers to the page navigation path during the process of the target terminal using the target client. For example, the page navigation path in some specific examples could be: launch the application → browse the community → click on a self-driving tour guide → bookmark → return to the homepage → open vehicle control, etc.

[0182] A directed graph is a graph structure where the edges have a direction. In the process of preprocessing page navigation paths into a directed graph, the business units associated with the page can be represented as nodes based on the obtained page navigation paths. Then, based on the navigation relationships between business units, the directed edges in the directed graph can be determined, thus obtaining the directed graph. A functional unit is the smallest unit that can independently carry out business functions and can be accessed or interacted with; it can be a page or a functional module within a page.

[0183] In some optional embodiments, the directed graph is a directed weighted graph, and the weight of the edge in the directed weighted graph is determined based on the page interaction metrics between the two business units connected by the edge.

[0184] A directed weighted graph refers to a directed graph in which edges are assigned weight values. In the relevant embodiments of this application, the weight of a directed edge is determined based on the page interaction metrics between the two business units connected by the directed edge.

[0185] A directed weighted graph in an application example can be represented as G=(V,E,W), where:

[0186] V is a set of nodes, representing functional units such as pages or functional modules of the target client, such as "Home", "Vehicle Control", "Community", "My", etc. The number of nodes can be set based on the number of functional units set in the target client. For example, the number of nodes in some examples is 50.

[0187] E is a set of edges. For example, (u→v) means jumping from functional unit u to functional unit v.

[0188] W represents the edge weight, indicating page interaction metrics such as the number of transitions or average dwell time.

[0189] Accordingly, the directed graph obtained based on the page navigation path is a directed weighted graph. The weights of the edges in the directed weighted graph are determined based on the page interaction metrics between the two business units connected by the edges. This allows for the comprehensive determination of the third feature vector by combining the page interaction metrics of the user object between the two business units. This helps to improve the accuracy of the obtained third feature vector, and further improves the accuracy of the obtained target member state features. In turn, it helps to improve the accuracy of the obtained member state, and also helps to further improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0190] The specific type of page interaction metrics is not limited. In some optional embodiments, page interaction metrics include at least one of the following: the number of jumps between two business units connected by an edge, and the average dwell time of the target member in the two business units.

[0191] Accordingly, the page interaction metrics used to determine the weights of edges in a directed weighted graph include at least one of the following: the number of jumps between two business units connected by an edge, and the average dwell time of a target member between the two business units connected by an edge. Since the number of jumps between two business units connected by an edge and the average dwell time between the two business units reflect the target member's interest in the business units and the relationship between them, the third feature vector can be comprehensively determined by combining these with the page interaction metrics of the target occupant between the two business units. This helps to improve the accuracy of the obtained third feature vector, and on this basis, it helps to further improve the accuracy of the obtained state characteristics of the target member, thereby improving the accuracy of the obtained member state, and also helps to further improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0192] The method of encoding the directed graph using a graph neural network to obtain the third feature vector is not limited. In some optional embodiments, refer to... Figure 6 As shown, a graph neural network is used to encode the directed graph to obtain the third feature vector, which includes:

[0193] Step 601: Based on the data information of the business units corresponding to the nodes in the directed graph, the nodes are encoded to obtain the node characteristics of each node. The data information includes at least one of the following: business unit type, business unit dwell time, and business unit bounce rate.

[0194] For the obtained directed graph, the nodes can be encoded based on the data information of the business units corresponding to each node to obtain the initial node characteristics. In some feasible embodiments:

[0195] For each business unit type (e.g., functional / content / settings / social), one-hot encoding can be applied to convert it into a binary encoded vector to obtain the business unit type characteristics.

[0196] Regarding the dwell time of business units, the dwell time of each business unit can be normalized, and after determining the dwell time range in which the dwell time of the business unit falls, the dwell time range can be mapped or one-hot encoded to obtain the dwell time characteristics of the business unit.

[0197] To determine the bounce rate of a business unit, one can obtain the bounce rate characteristics of each business unit by performing linear projection or multi-layer perception processing on the bounce rate. Alternatively, one can determine the bounce rate interval in which the business unit bounce rate falls and then perform mapping or one-hot encoding on the bounce rate interval to obtain the bounce rate characteristics of the business unit.

[0198] By combining features such as the business unit type, dwell time, and bounce rate of a node, initial node features can be obtained through fusion methods such as splicing. It is understood that in other implementations, initial node features can also be obtained through other methods.

[0199] Step 602: Perform graph attention layer processing on the node features of each node to obtain the updated directed graph.

[0200] Based on the node features obtained for each node, a graph attention layer can be applied to perform attention-weighted aggregation of the node features to obtain the updated node features, thereby obtaining the updated directed graph.

[0201] Step 603: Perform global pooling on the updated directed graph to obtain the third feature vector.

[0202] The updated directed graph can be processed by global pooling to obtain the corresponding third feature vector.

[0203] In the relevant examples of this application, a directed graph can be encoded using a Graph Attention Network (GAT) to obtain a third feature vector. The GAT performs node feature initialization (i.e., encoding) on ​​the nodes to obtain the node features of each node.

[0204] The Graph Attention Network (GAT) processes the node features of each node using a graph attention layer to obtain an updated directed graph. In some specific examples, the graph attention layer may include two layers, each with eight attention heads. The graph attention layer processing in some examples can be expressed by the following formula:

[0205] .

[0206] in, For nodes In the Layer representation, For nodes In the Vector representation of layers, For activation function, For nodes The set of neighboring nodes, For nodes , For learnable linear transformations, This represents the attention coefficient.

[0207] In some feasible embodiments, attention coefficient It can be determined in the following ways:

[0208] .

[0209] in, For nodes Normalization of all neighboring nodes, It is a non-linear activation function. Let be the attention parameter vector, which contains learnable parameters. For learnable linear transformations, For nodes In the Vector representation of layers, For nodes In the The vector representation of a layer, where || denotes vector concatenation;

[0210] Based on the updated directed graph obtained through the graph attention layer, global average pooling can be performed using the graph pooling layer of the Graph Attention Network (GAT) to obtain the full graph representation, which can be expressed by the formula:

[0211] ; .

[0212] in, For average pooling function, For the Lth (last) layer node The vector representation of , This is the third feature vector obtained after global average pooling. To make the third feature vector The third feature vector after mapping to a preset dimension (e.g., 64 dimensions), It is a multilayer perceptron.

[0213] Accordingly, in the process of encoding a directed graph to obtain the third feature vector, the nodes are initially encoded based on data such as the business unit type, business unit dwell time, and business unit bounce rate of each node in the directed graph. After obtaining the node features of each node, the node features of each node are processed by a graph attention layer to obtain an updated directed graph. Then, the updated directed graph is processed by global pooling to obtain the third feature vector. This helps to improve the accuracy of the obtained third feature vector. Based on this, it helps to further improve the accuracy of the obtained target member state features, thereby improving the accuracy of the obtained member state. It also helps to improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0214] The method of acquiring in-vehicle perception data corresponding to target occupants is not limited; in some optional embodiments, refer to... Figure 7As shown, step 201, obtaining in-vehicle perception data corresponding to the target occupants, includes:

[0215] Step 701: Obtain at least one type of in-vehicle interaction data generated by the interaction between the target member and the vehicle;

[0216] Step 702: Acquire physiological state data of the target member collected through wearable devices;

[0217] Step 703: Determine the in-vehicle perception data corresponding to the target member based on at least one of the in-vehicle interaction data and physiological state data.

[0218] In-vehicle interaction data refers to the interaction data generated during human-computer interaction between an object and a vehicle.

[0219] Accordingly, in the process of acquiring in-vehicle perception data corresponding to the target occupant, the data is determined based on at least one of the in-vehicle interaction data generated by the interaction between the target occupant and the vehicle, and the physiological state data of the target occupant collected through wearable devices. Since the target occupant interacts with the vehicle based on their potential personalized behaviors, by using at least one in-vehicle interaction data generated by the interaction between the target occupant and the vehicle as in-vehicle perception data, personalized interaction information generated by the interaction between the target occupant and the vehicle can be obtained. This helps to improve the accuracy of the obtained state characteristics of the target occupant, and thus helps to improve the accuracy of the obtained occupant state. Physiological state data reflects the physiological state of the target occupant. Therefore, by using the physiological state data of the target occupant as in-vehicle perception data, the state characteristics of the target occupant can be obtained by combining the physiological state data of the target occupant. On this basis, when determining the occupant state based on the state characteristics of the target occupant and providing personalized proactive services accordingly, it helps to combine the physiological state of the target occupant to provide personalized proactive services, which helps to improve the accuracy of the obtained occupant state and also helps to improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0220] The specific data type of the acquired in-vehicle interaction data is not limited. In some optional embodiments, the in-vehicle interaction data includes at least one of touch data, voice data, and physical operation data. Among them, touch data includes at least one of touch coordinates, touch pressure value, touch area, and touch click interval; voice data includes at least one of voice command text, acoustic features, and tone energy; and physical operation data includes at least one of the rotation angle of physical knobs, the pressing depth of physical buttons, and the change in steering wheel grip force.

[0221] Accordingly, when obtaining in-vehicle interaction data, different types of data can be obtained, such as touch data, voice data, and physical operation data. Touch data, in addition to touch coordinates, can also include touch pressure values, touch area, and touch click intervals. Voice data, in addition to voice command text, can also include acoustic features and tone energy. Physical operation data can simultaneously include the rotation angle of physical knobs, the pressing depth of physical buttons, and changes in steering wheel grip force. The touch pressure value, touch area, touch click interval, acoustic features, tone energy, the rotation angle of physical knobs, the pressing depth of physical buttons, and changes in steering wheel grip force are closely related to the personalized characteristics of the target occupant. Therefore, the accuracy of the target occupant's state characteristics can be further obtained based on this. On this basis, when determining the occupant's state based on the target occupant's state characteristics and providing personalized proactive services accordingly, it helps to improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0222] There are no limitations on the method for feature extraction from in-vehicle perception data. In some optional embodiments, refer to... Figure 8 As shown, when the in-vehicle perception data includes in-vehicle interaction data, step 201, which involves feature extraction from the in-vehicle perception data, includes:

[0223] Step 801: In response to in-vehicle perception data, including in-vehicle interaction data, based on the data type of the in-vehicle interaction data, encode it using the corresponding encoding method to obtain the encoded sub-features corresponding to each data type.

[0224] Different encoding methods can be used to encode in-vehicle interaction data of different data types in order to obtain the corresponding encoded sub-features for each data type.

[0225] In some optional embodiments, taking in-vehicle interaction data including touch data, voice data, and physical operation data as an example, the obtained coded sub-features may include touch sub-features, voice sub-features, and operation sub-features. In this case, refer to... Figure 9 As shown, the data types based on in-vehicle interaction data in step 801 are encoded using corresponding encoding methods to obtain the encoded sub-features corresponding to each data type, which may include:

[0226] Step 901: Determine the touch sequence based on the touch data, and encode the touch sequence to obtain touch sub-features.

[0227] The method of determining touch sequences based on touch data and encoding the touch sequences is not limited. In some examples, the collected touch data can be preprocessed, sorted by time sequence, and independent touch actions can be distinguished according to time intervals, such as click, long press, and swipe. The order of occurrence of each touch action can be arranged to form a complete and orderly touch behavior sequence. Then, continuous features such as touch position, force, duration, and direction can be converted into discrete level labels. Through the established mapping relationship between touch features and discrete features, the touch sequence can be converted into a standardized digital code sequence to obtain touch sub-features. However, this method is not limited to this.

[0228] Step 902: Extract prosodic features from the speech data, obtain the corresponding speech text, extract semantic features from the speech text, fuse semantic features and prosodic features to obtain speech sub-features.

[0229] Prosodic features of speech data refer to characteristics such as tone, rhythm, stress, speed, and pitch. Different users exhibit different prosodic features when speaking, and the same user will also exhibit different prosodic features under different emotional states. For example, the rhythm, stress, and speed of speech will vary. Therefore, by extracting prosodic features from speech data and integrating them with the semantic features of the corresponding speech text, the obtained speech sub-features can reflect not only the semantic information of the speech data but also the personalized characteristics of the user, such as emotions and states, which can help improve the accuracy of the obtained speech sub-features.

[0230] Step 903: Perform feature statistics on the physical operation data to obtain at least one statistical feature for each physical operation, and fuse the statistical features of each physical operation to obtain operation sub-features.

[0231] The specific statistical types for feature statistics of physical operation data are not limited, including but not limited to mean, variance, peak value, valley value, etc. The method of fusing the statistical features of each physical operation to obtain operation sub-features is not limited; for example, the obtained statistical features of each physical operation can be concatenated to obtain operation sub-features.

[0232] Accordingly, when in-vehicle interaction data includes touch data, voice data, and physical operation data, for touch data, after determining the touch sequence based on the touch data, the touch sub-features are obtained by sequence encoding the touch sequence. For voice data, on the one hand, prosodic features of the voice data are extracted; on the other hand, after obtaining the corresponding voice text, semantic features of the voice text are extracted, and then the semantic features and prosodic features are fused to obtain the voice sub-features corresponding to the voice data. Thus, voice sub-features can be obtained by combining two different dimensions of voice prosody and voice content. For physical operation data, feature statistics are performed to obtain at least one statistical feature for each physical operation, and the statistical features are fused to obtain operation sub-features. Thus, operation sub-features can be obtained by combining the statistical characteristics of different dimensions of physical operations, which can improve the accuracy of the obtained touch sub-features, voice sub-features, and operation sub-features. This helps to improve the accuracy of the obtained target member's state features. On this basis, when determining the member's state based on the target member's state features and providing personalized proactive services accordingly, it can help improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0233] Step 802: Perform attention fusion processing on the encoded sub-features corresponding to each data type to obtain the vehicle behavior feature vector, the first feature including the vehicle behavior feature vector.

[0234] In some feasible embodiments, the in-vehicle behavior feature vector can be obtained by fusing the encoded sub-features corresponding to each data type obtained by a multimodal fusion encoder. The multimodal fusion encoder may include a touch sequence processing branch, a semantic data processing branch, a physical operation data processing branch, and a cross-branch fusion processing layer.

[0235] The touch sequence processing branch encodes the touch sequence to obtain touch sub-features. In some examples, this touch sequence processing branch may include a Transformer encoder. By inputting the touch sequence into the Transformer encoder, the corresponding touch sub-features can be obtained. In some specific examples, the Transformer encoder may be a 2-layer 4-head encoder, but it is not limited to this.

[0236] The semantic data processing branch extracts prosodic features from the speech data, obtains the corresponding speech text, and extracts semantic features from the speech text. It then fuses the semantic and prosodic features to obtain speech sub-features. In some examples, the semantic data processing branch can use MiniLM (a lightweight Transformer model designed for distillation) to extract semantic features from the speech text, and CNN (Convolutional Neural Network) to extract prosodic features from the speech data. The semantic and prosodic features are then fused, for example, through LayerNorm (Layer Normalization), to obtain speech sub-features. In some specific examples, the MiniLM model can be a 6-layer, 384-dimensional model. Speech sub-features in some application examples can be expressed by the following formula:

[0237] .

[0238] in, For speech sub-features, For layer normalization processing, For semantic features, It is a rhythmic feature;

[0239] The physical operation data processing branch is used to perform feature statistics on physical operation data, obtain at least one statistical feature of each physical operation, such as mean, variance, peak value, valley value, etc., and fuse the statistical features of each physical operation to obtain operation sub-features.

[0240] The cross-branch fusion processing layer is used to perform attention fusion processing on touch sub-features, voice sub-features and operation sub-features. It dynamically fuses touch sub-features, voice sub-features and operation features through a cross-branch cross-attention mechanism to obtain vehicle behavior feature vectors.

[0241] Accordingly, in the process of extracting features from in-vehicle interaction data to obtain in-vehicle behavior feature vectors, the data types of in-vehicle interaction data are encoded using corresponding encoding methods to obtain their respective coded sub-features. Then, attention fusion processing is performed on the coded sub-features corresponding to each data type to obtain the in-vehicle behavior feature vector. Thus, in-vehicle perception data of different data types can be encoded using different encoding methods, and a more suitable encoding method can be used. This is applicable to the data characteristics of different data types and helps to improve the accuracy of the obtained coded sub-features. On this basis, further attention fusion processing can fully utilize the complementary characteristics of coded sub-features of different data types, which helps to improve the accuracy and generalization ability of the obtained in-vehicle behavior feature vectors.

[0242] There are no restrictions on the methods for obtaining physiological status data. In some feasible implementations, physiological status data of the target member can be obtained in real time from wearable devices such as smartwatches and wristbands via Bluetooth, or through wearable device apps. However, it is not limited to these methods, as long as the target member's physiological status data is available.

[0243] The specific data type of the acquired physiological state data is not limited. In some optional embodiments, the physiological state data includes at least one of heart rate, heart rate variability, skin conductance response, and skin temperature.

[0244] Accordingly, the obtained physiological state data can include different physiological state data such as heart rate, heart rate variability, skin conductance response, and skin temperature. This allows the physiological state of the target member to be obtained from different dimensions. Based on this, when determining the member's state based on the target member's state characteristics and providing personalized proactive services accordingly, it can help to provide personalized proactive services by combining different types of physiological states of the target member. This will help improve the matching degree and accuracy of providing personalized proactive services in the intelligent cockpit environment.

[0245] In some alternative embodiments, reference is made to Figure 10 As shown, when the in-vehicle perception data includes physiological state data, step 201, which involves feature extraction from the in-vehicle perception data, includes:

[0246] Step 1001: In response to in-vehicle perception data, including physiological state data, the physiological state data is encoded using convolutional kernels of different scales to obtain multiple physiological features at different scales.

[0247] The specific settings for the size of convolutional kernels at different scales are not limited. Some examples may use kernels of 3, 7, 15, etc., but are not limited to these. Furthermore, the number of output channels for convolutional kernels of different scales can be the same, for example, all can have 32 channels.

[0248] In related implementations, the original physiological state data may be filtered, denoised, and normalized, and then preprocessed by sliding window segmentation according to a fixed duration (e.g., 120 seconds) and step size (e.g., 30 seconds) to obtain preprocessed physiological state data. The preprocessed physiological state data may be encoded by convolutional kernels of different scales. The specific preprocessing method is not limited in this application embodiment.

[0249] Step 1002: Perform splicing on multiple physiological features at different scales to obtain spliced ​​features.

[0250] Physiological features obtained at different scales can be spliced ​​together to obtain spliced ​​features.

[0251] Step 1003: Perform temporal attention pooling on the concatenated features to obtain the physiological state feature vector, the first feature including the physiological state feature vector.

[0252] The obtained concatenated features can be processed using time-series attention pooling to obtain physiological state feature vectors.

[0253] In relevant examples, a physiological state feature vector can be obtained by extracting features from physiological state data using a multi-scale one-dimensional convolutional neural network. This multi-scale one-dimensional convolutional neural network sets up multiple convolutional kernels of different sizes (such as 3, 7, and 15) to capture physiological features that change rapidly (such as heartbeat details) and slowly (such as respiratory rhythm), respectively. The outputs of the branches of the three convolutional layers (i.e., the physiological features at different scales mentioned above) are concatenated and subjected to temporal attention pooling to obtain a physiological state feature vector.

[0254] Accordingly, in the process of extracting features from physiological state data and obtaining physiological state feature vectors, the physiological state data is encoded by convolutional kernels of different scales to obtain multiple physiological features of different scales. These features are then concatenated to obtain concatenated features. Temporal attention pooling is then applied to the concatenated features to obtain the physiological state feature vector. By first encoding the physiological state data with convolutional kernels of different scales, both rapidly changing and slowly changing physiological features can be captured. The subsequent temporal attention pooling process, which fully utilizes the complementary characteristics of rapidly changing and slowly changing physiological features, helps to improve the accuracy and generalization ability of the obtained physiological state feature vector.

[0255] Based on the first and second features, the method of feature fusion processing to obtain the member state is not limited. In some optional embodiments, the first feature includes more than one feature vector, for example, the first feature includes the vehicle behavior feature vector and physiological state feature vector as described above. The second feature includes more than one feature vector, for example, the second feature includes the first feature vector corresponding to the target website, the second feature vector corresponding to the target mini-program, and the third feature vector corresponding to the target client as described above. (Reference) Figure 11 As shown, feature fusion processing is performed on the first feature and the second feature to obtain the state features of the target member, including:

[0256] Step 1101: Construct an input matrix based on the first and second features. Each row of the input matrix represents a feature vector.

[0257] Based on the obtained first and second features, namely the vehicle behavior feature vector, physiological state feature vector, first feature vector, second feature vector, and third feature vector obtained above, an input matrix can be constructed. The vehicle behavior feature vector, physiological state feature vector, first feature vector, second feature vector, and third feature vector can be used as five "tokens" and combined into an input matrix.

[0258] In some examples, the obtained input matrix can be expressed by the formula:

[0259] .

[0260] in, The first eigenvector, This is the second feature vector. The third eigenvector, This is a feature vector representing the physiological state. This is the vehicle behavior feature vector.

[0261] Among them, the vehicle behavior feature vector, physiological state feature vector, first feature vector, second feature vector, and third feature vector can be feature vectors of the same dimension, such as 64-dimensional feature vectors.

[0262] In the relevant embodiments of this application, learnable modal position codes can be added to the vehicle behavior feature vector, physiological state feature vector, first feature vector, second feature vector, and third feature vector, respectively. These modal position codes are then added to the tokens corresponding to the feature vectors to form an input matrix for subsequent processing. Accordingly, the input matrix with added learnable modal position codes can be represented as: ,in, This is a learnable modal position encoding matrix.

[0263] Step 1102: Based on the input matrix, calculate the attention weights between any two rows in the input matrix to obtain the attention weight matrix.

[0264] There are no restrictions on how the attention weights between any two rows in the input matrix can be calculated. For example, a multi-head self-attention mechanism can be used to calculate the attention weights between any two rows, thereby obtaining the attention weights between the two modal features corresponding to the two rows, forming an attention weight matrix. This attention weight matrix reflects the correlation strength between the two different modal features corresponding to the two rows under the current input. For example, the high heart rate feature in the physiological state feature vector and the frequent voice commands in the vehicle behavior feature vector will have a relatively high attention weight.

[0265] Step 1103: Multiply the input matrix and the attention weight matrix to obtain the state features of the target member.

[0266] Based on the obtained attention weight matrix, the input matrix and attention weight matrix are multiplied to achieve adaptive weighted summation of the features corresponding to each row of the input matrix. This allows us to obtain the state features of the target member based on the attention mechanism, which integrates key information from various modalities. This state feature is a global feature vector.

[0267] Accordingly, in the process of fusing the first feature and the second feature to obtain the state features of the target member, and determining the member state of the target member based on the state features, the first feature and the second feature are constructed as an input matrix. Then, the attention weight matrix is ​​obtained by calculating the attention weight between any two rows in the input matrix. After multiplying the attention weight matrix and the input matrix to obtain the state features of the target member, the state features are mapped by a fully connected layer to obtain the member state of the target member. That is, in the process of fusing the first feature and the second feature, the cross-modal attention fusion mechanism can fully utilize the complementary characteristics of different features, which can help improve the accuracy of the obtained state features of the target member.

[0268] There are no limitations on the way to determine the member state of the target member based on state features. In some feasible embodiments, the state features can be mapped by a fully connected layer to obtain the member state of the target member.

[0269] Based on the obtained global state features, these features can be input into one or more fully connected layers (i.e., state mapping layers) to map them into a final user state vector, thereby obtaining the member state of the target member. This user state vector can be a K-dimensional (K≥2) continuous floating-point array, where each dimension represents a preset state indicator. This indicator can be a psychological or behavioral indicator, such as fatigue level (0-1 continuous value), emotional valence (-1 to +1), focus level, intensity of operational intention, etc., but is not limited to these.

[0270] In some feasible embodiments, based on the input matrix constructed based on the first and second features, a Transformer encoder with a multi-head self-attention mechanism can be used to calculate the attention weights between any two rows (i.e., two features) in the input matrix, obtaining an attention weight matrix. Based on this attention weight matrix, a weighted summation is performed on each feature in the input matrix to obtain the state features of the target member. Finally, a fully connected layer is applied to the state features to obtain the member state of the target member. Specific processing methods in related examples are discussed in conjunction with... Figure 15 As shown, it can be described as follows:

[0271] First, input features (i.e., the input matrix of the learnable modal position encoding mentioned above) ) respectively with the learnable weight matrix , , Perform a linear transformation to obtain the query matrix Q, the key matrix K, and the sum matrix V, where, , , .

[0272] Then, multi-head attention calculations are performed on the query matrix Q, key matrix K, and sum matrix V. In some examples, this can be done by splitting the query matrix Q, key matrix K, and sum matrix V into four attention heads, each with a fixed dimension of 16. For example, the similarity matrix between the query matrix Q and the key matrix K can be calculated first, and then divided by... After scaling and softmax normalization to obtain attention weights, these weights are then weighted and summed with the corresponding head's value matrix V to obtain the attention output for each head. This can be expressed by the formula:

[0273] .

[0274] in, It is the first Attention output based on size For the first The query matrix Q of the size, For the first The key matrix K of the size, For the first V is the value matrix of the individual heads. The scaling factor is determined based on a dimension of 16 for each attention head. If the dimension of each attention head is other than 16, the scaling factor can also be other values.

[0275] Subsequently, multi-head stitching and projection are performed, that is, the results of the four attention heads can be stitched back to the original 64 dimensions according to the dimensions, and then learned weights are used. Perform linear projection to obtain the complete multi-head attention output. It can be expressed by the formula:

[0276] .

[0277] in, For splicing processing, The attention output of the first attention head. The attention output of the 4th attention head. is a learnable parameter matrix.

[0278] Then, input features With complete multi-head attention output After summing the residuals, layer normalization is performed (i.e., summation followed by layer normalization) to obtain the features processed by the attention module. It can be expressed by the formula: .

[0279] Features obtained after attention module processing Further processing involves FFN (Feed-Forward Network) and secondary residual normalization. In relevant examples, this could involve processing the features after the attention module. The data is fed into a two-layer feedforward network: first, it is increased to 256 dimensions through a first-layer linear transformation, activated by the ReLU activation function, and then reduced back to 64 dimensions through a second-layer linear transformation to obtain the features output by the FFN. ; FFN output features and The residuals are summed, and then layer normalization is performed again to obtain the final sequence features of the encoder. It can be expressed by the formula:

[0280] ;

[0281] .

[0282] in, The weights are the first-level linear transformation weights of the FFN. This is the bias for the first linear transformation of FFN. The weights are the weights of the second-level linear transformation of the FFN. This is the bias for the second-level linear transformation of FFN. It is a non-linear activation function. This is for layer normalization processing.

[0283] Subsequently, the final sequence features obtained were analyzed. Global pooling is performed to obtain the state features of the target member. The specific method of global pooling is not limited; in some feasible embodiments, it can be achieved by average pooling of the final output sequence features of the five tokens (i.e., corresponding feature vectors). The average is calculated to obtain a vector of global state features, which can be expressed by the formula: In some other feasible embodiments, the final output sequence feature corresponding to the token whose sequence is the first can be directly taken. The output is a vector representing the global state features, but is not limited to this.

[0284] Based on the obtained global state features, a fully connected layer can be applied to map these features to obtain the member state of the target member. In some feasible implementations, a learnable linear transformation can be performed on the global state features, followed by nonlinear activation using a nonlinear activation function to obtain the member state of the target member. This can be expressed by the following formula: .in, For the member status of the target member, It is a non-linear activation function. This is the weight matrix. For bias.

[0285] The output of the target member's status can include multiple dimensions, each representing a continuous status index. For example, the first dimension can represent fatigue level (e.g., 0 indicates alertness, 1 indicates extreme fatigue), the second dimension can represent emotional valence (e.g., -1 indicates negativity, +1 indicates positivity), the third dimension can represent focus level (e.g., 0 indicates distraction, 1 indicates high focus), and the fourth dimension can represent operational intention (e.g., 0 indicates no operational intention, 1 indicates strong operational intention). Dimensions 5 to 8 can be reserved to support personalized configurations, such as tension level, familiarity level, etc., but are not limited to these.

[0286] Based on the embodiments described above, the following is an example of a specific application.

[0287] The vehicle occupant data processing method provided in this application can acquire heterogeneous data from different modalities, including external digital footprints (such as data from websites, mini-programs, and clients), real-time in-vehicle interaction data, and physiological state data. It can also effectively integrate heterogeneous data from different modalities to construct and perceive a comprehensive user status, thereby providing an information foundation for providing more comprehensive and matching personalized proactive services in the intelligent cockpit environment.

[0288] In the implementation of the method in the specific example of this application, a multimodal fusion model can be provided. This multimodal fusion model may include setting independent feature encoders for five different data sources (i.e., data from the site, data from the lightweight application, data from the client application, real-time in-vehicle interaction data, and physiological state data) to extract high-dimensional feature representations specific to each modality of the different data sources. Then, through a cross-modal attention fusion layer, the dynamic correlation weights between features from different data sources are learned to achieve adaptive fusion. Finally, a dynamic vector that can represent the user's current comprehensive state, i.e., the user's member state, is generated. This member state can be a continuous and multidimensional dynamic vector.

[0289] In some feasible embodiments, the method further includes:

[0290] Obtain the historical state characteristics of the target member;

[0291] The state features of the target member and its historical state features are smoothed to obtain the smoothed output state features.

[0292] Output characteristics after smoothing.

[0293] In some examples, the historical state characteristics of the target member can refer to state characteristics generated in real time based on data from the data source at one or more historical moments. In other examples, it can refer to the state characteristics of the target member output at one or more historical moments. That is, the state characteristics can be state characteristics that have already undergone smoothing. The method of smoothing the state characteristics and historical state characteristics of the target member is not limited, such as weighted fusion, smoothing based on exponential moving average, or adaptive smoothing according to the degree of jump. This application does not impose specific restrictions on the specific method of smoothing.

[0294] Accordingly, the output state features of the target member are the state features after smoothing by combining the historical state features of the target member. This achieves state following based on time sequence, which helps to solve the problem of abnormal jumps in the obtained state features and is more in line with the objective law of continuous changes in user state.

[0295] The overall architecture of the embodiments in this application is as follows: Figure 12 As shown, it includes a data acquisition and preprocessing module 1201, a multimodal feature encoding module 1202, a cross-modal attention fusion module 1203, and a state vector output module 1204. In some feasible embodiments, a time-series smoothing module (not shown in the figure) can also be provided to smooth the state features of the target member and the historical state features of the target member, wherein:

[0296] The data acquisition and preprocessing module 1201 is used to collect data from multiple different data sources, including: interaction log data from the target website, interaction feedback data from the target mini-program, page jump path from the target client, physiological state data, and in-vehicle interaction data, and to perform corresponding data preprocessing on the different data sources collected.

[0297] The multimodal feature encoding module 1202 includes multiple different encoders, such as Figure 12 The diagram shows the architecture of the five independent encoders included in the multimodal feature encoding module 1202: the site data encoder, the lightweight application data encoder, the map encoder, the physiological data encoder, and the behavior encoder. Figure 13 As shown, combined with Figure 13 visible:

[0298] For the interaction log data of the target website, the site data encoder performs sequence encoding on the preprocessed interaction log data, and then obtains the first feature vector through self-attention pooling. ;

[0299] For the interactive feedback data of the target mini-program, the lightweight application data encoder extracts semantic features from the text data and behavioral features from the preprocessed interactive feedback data. The semantic and behavioral features are then fused to obtain a second feature vector. ;

[0300] For the page navigation path of the target client, the graph encoder initializes each node in the directed graph corresponding to the page navigation path to obtain node features. Then, it processes the updated directed graph through a graph attention network layer and further performs global pooling to obtain the third feature vector. ;

[0301] For physiological state data, the physiological data encoder encodes the preprocessed physiological state data using convolutional kernels of different scales to obtain multiple physiological features at different scales. These features are then concatenated to obtain concatenated features, which are then subjected to temporal attention pooling to obtain the physiological state feature vector. ;

[0302] For in-vehicle interaction data, the behavior encoder generates touch sequences from the touch data and performs sequence encoding on these sequences to obtain touch sub-features. It also extracts prosodic features from the speech data within the in-vehicle interaction data and extracts semantic features from the corresponding speech text. These semantic and prosodic features are then fused to obtain speech sub-features. Finally, the encoder performs feature statistics and fusion on the physical operation data within the in-vehicle interaction data to obtain operation sub-features. The touch, speech, and operation sub-features are then fused, for example, through cross-branch attention fusion, to obtain the in-vehicle behavior feature vector. .

[0303] The cross-modal attention fusion module 1203 performs cross-modal attention fusion processing on the features output by each encoder of the multimodal feature encoding module 1202, including the first feature vector, the second feature vector, the third feature vector, the physiological state feature vector, and the vehicle behavior feature vector, to obtain the state features of the target member. The state vector output module 1204 then maps the global state features to obtain the member state of the target member.

[0304] The deployment architecture of this embodiment can be as follows: Figure 14 As shown, the cloud server 1401 can obtain interaction log data of the target website from the web server 1402, in-vehicle interaction data from the vehicle terminal (smart cockpit) 1404, and interaction feedback data of the target mini-program generated by interaction through the mobile terminal 1403, the page jump path of the target client, and physiological state data detected by the wearable device obtained by the wearable device application from the mobile terminal 1403 or a backend server associated with the mobile terminal 1403 (not shown in the figure). The cloud server 1401 can train each encoder involved in the multimodal feature encoding module 1202 based on historically collected data to obtain trained encoders. The trained encoders can be deployed to the relevant execution devices in this embodiment, such as the vehicle terminal 1404 or a server. The specific method for training the encoders can be any method already existing in the related technology; this embodiment does not impose specific limitations on this. The vehicle terminal 1404 can be equipped with a lightweight encoder to extract features from the data from the five different data sources mentioned above, perform multi-head attention fusion processing, and then output the user's membership status.

[0305] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0306] Based on the same inventive concept, this application also provides a vehicle occupant data processing apparatus for implementing the above-described method for processing vehicle occupant data. The solution provided by this apparatus is similar to the implementation described in the above-described method; therefore, the specific limitations in one or more embodiments of the vehicle occupant data processing apparatus provided below can be found in the limitations of the vehicle occupant data processing method described above, and will not be repeated here.

[0307] In one exemplary embodiment, such as Figure 16 As shown, a vehicle occupant data processing device is provided, including: an in-vehicle feature acquisition module 1601, a channel interaction feature acquisition module 1602, and a feature fusion module 1603, wherein:

[0308] The in-vehicle feature acquisition module 1601 is used to acquire in-vehicle perception data corresponding to the target occupants in the vehicle, and to extract features from the in-vehicle perception data to obtain the first feature;

[0309] The channel interaction feature acquisition module 1602 is used to acquire the interaction data of the target member in at least one interaction channel, and to extract features from the interaction data to obtain a second feature;

[0310] The feature fusion module 1603 is used to perform feature fusion processing on the first feature and the second feature to obtain the state features of the target member, and to determine the member state of the target member based on the state features.

[0311] In some optional embodiments, the interaction data includes at least one of the following: interaction log data obtained from the target website, interaction feedback data obtained from the target mini-program, and page jump path obtained from the target client.

[0312] In some optional embodiments, the channel interaction feature acquisition module 1602 is configured to: generate a user behavior sequence based on the interaction log data in response to interaction data including interaction log data, and perform sequence encoding on the user behavior sequence to obtain a first feature vector; perform feature encoding on the interaction feedback data in response to interaction data including interaction feedback data to obtain a second feature vector; preprocess the page jump path into a directed graph in response to interaction data including page jump path, wherein nodes in the directed graph represent business units in the target client, and edges in the directed graph represent jump relationships between two connected business units; encode the directed graph using a graph neural network to obtain a third feature vector; and determine the second feature based on at least one of the first feature vector, the second feature vector, and the third feature vector.

[0313] In some optional embodiments, the channel interaction feature acquisition module 1602 is used to extract features from the text data in the interaction feedback data to obtain a semantic feature vector; extract features from the behavioral data in the interaction feedback data to obtain a behavioral feature vector; and combine the semantic feature vector and the behavioral feature vector to obtain a second feature vector.

[0314] In some optional embodiments, the in-vehicle feature acquisition module 1601 is used to acquire at least one in-vehicle interaction data generated by the interaction between the target member and the vehicle; acquire physiological state data of the target member collected by a wearable device; and determine the in-vehicle perception data corresponding to the target member based on at least one of the in-vehicle interaction data and the physiological state data.

[0315] In some optional embodiments, the in-vehicle feature acquisition module 1601 is used to respond to in-vehicle perception data, including in-vehicle interaction data, and to encode the in-vehicle interaction data according to the corresponding encoding method based on the data type of the in-vehicle interaction data to obtain the encoded sub-features corresponding to each data type; and to perform attention fusion processing on the encoded sub-features corresponding to each data type to obtain the in-vehicle behavior feature vector, wherein the first feature includes the in-vehicle behavior feature vector.

[0316] In some optional embodiments, the in-vehicle feature acquisition module 1601 is used to encode the physiological state data by using convolutional kernels of different scales in response to in-vehicle perception data including physiological state data, to obtain multiple physiological features of different scales; to concatenate the multiple physiological features of different scales to obtain concatenated features; and to perform temporal attention pooling on the concatenated features to obtain a physiological state feature vector, wherein the first feature includes the physiological state feature vector.

[0317] In some optional embodiments, the in-vehicle interaction data includes: touch data, voice data, and physical operation data; the encoded sub-features include touch sub-features, voice sub-features, and operation sub-features; the in-vehicle feature acquisition module 1601 is used to determine the touch sequence based on the touch data, and to perform sequence encoding on the touch sequence to obtain touch sub-features; extract the prosodic features of the voice data, obtain the voice text corresponding to the voice data, and extract the semantic features of the voice text, and fuse the semantic features and prosodic features to obtain voice sub-features; perform feature statistics on the physical operation data, obtain at least one statistical feature of each physical operation, and fuse the statistical features of each physical operation to obtain operation sub-features.

[0318] In some optional embodiments, the first feature includes more than one feature vector, and the second feature includes more than one feature vector; the feature fusion module 1603 is used to construct an input matrix based on the first feature and the second feature, where each row of the input matrix represents a feature vector; based on the input matrix, the attention weights between any two rows in the input matrix are calculated to obtain an attention weight matrix; the input matrix and the attention weight matrix are multiplied to obtain the state features of the target member; and the state features are mapped by a fully connected layer to obtain the member state of the target member.

[0319] Each module in the aforementioned vehicle occupant data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0320] In one exemplary embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0321] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0322] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0323] It should be noted that all information and data involved in this application (including but not limited to data used for analysis, stored data, and displayed data) are information and data authorized by the user or fully authorized by all parties, and the acquisition, transmission, storage, use, and processing of the relevant data comply with the relevant provisions of laws and regulations. Users can refuse or easily refuse content pushed to them (e.g., personalized recommendation information). In the embodiments of this application, certain existing solutions in the industry, such as software, components, and models, may be mentioned. These should be considered exemplary, and their purpose is merely to illustrate the feasibility of implementing the technical solution of this application, but does not mean that the applicant has already used or necessarily used such a solution.

[0324] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program mentioned can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0325] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0326] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing vehicle occupant data, characterized in that, The method includes: The in-vehicle perception data corresponding to the target member in the vehicle is obtained, and the in-vehicle perception data is used to extract features to obtain a first feature. The in-vehicle perception data includes at least one of the following: at least one in-vehicle interaction data generated by the target member interacting with the vehicle, and at least one of the physiological state data of the target member collected by a wearable device. The first feature includes more than one feature vector. The interaction data of the target member in at least one interaction channel is obtained, and the interaction data is used to extract features to obtain a second feature. The interaction data includes at least one of the following: interaction log data obtained from the target website, interaction feedback data obtained from the target mini program, and page jump path obtained from the target client. The second feature includes one or more feature vectors. An input matrix is ​​constructed based on the first feature and the second feature, where each row of the input matrix represents a feature vector. Based on the input matrix, the attention weights between any two rows in the input matrix are calculated to obtain an attention weight matrix. The input matrix and the attention weight matrix are multiplied to obtain the state features of the target member. The state features are then mapped to obtain the member state of the target member. The step of extracting features from the interaction data to obtain a second feature includes: In response to the interaction data including the interaction log data, a user behavior sequence is generated based on the interaction log data, and the user behavior sequence is sequence encoded to obtain a first feature vector; In response to the interaction data including interaction feedback data, feature encoding is performed on the interaction feedback data to obtain a second feature vector; In response to the interaction data including the page jump path, the page jump path is preprocessed into a directed graph, wherein the nodes in the directed graph represent business units in the target client, and the edges in the directed graph represent the jump relationship between two connected business units; a graph neural network is used to encode the directed graph to obtain a third feature vector; The second feature is determined based on at least one of the first feature vector, the second feature vector, and the third feature vector.

2. The method according to claim 1, characterized in that, The step of encoding the interactive feedback data to obtain a second feature vector includes: Feature extraction is performed on the text data in the interactive feedback data to obtain semantic feature vectors; Feature extraction is performed on the behavioral data in the interactive feedback data to obtain a behavioral feature vector; The second feature vector is obtained by combining the semantic feature vector and the behavioral feature vector.

3. The method according to claim 1 or 2, characterized in that, The acquisition of in-vehicle perception data corresponding to the target occupants includes: Acquire at least one type of in-vehicle interaction data generated by the interaction between the target member and the vehicle; Acquire physiological state data of the target member collected through wearable devices; The in-vehicle perception data corresponding to the target member is determined based on at least one of the in-vehicle interaction data and the physiological state data.

4. The method according to claim 3, characterized in that, The feature extraction of the in-vehicle perception data includes: In response to the in-vehicle perception data including the in-vehicle interaction data, the data is encoded using a corresponding encoding method based on the data type of the in-vehicle interaction data to obtain the encoded sub-features corresponding to each data type. Attention fusion processing is performed on the encoded sub-features corresponding to each of the data types to obtain the vehicle behavior feature vector, wherein the first feature includes the vehicle behavior feature vector.

5. The method according to claim 4, characterized in that, The feature extraction of the in-vehicle perception data includes: In response to the in-vehicle perception data including the physiological state data, the physiological state data is encoded by convolutional kernels of multiple scales to obtain physiological features corresponding to multiple scales; The physiological features corresponding to multiple scales are spliced ​​together to obtain the spliced ​​features; The concatenated features are subjected to temporal attention pooling to obtain a physiological state feature vector, wherein the first feature includes the physiological state feature vector.

6. The method according to claim 4, characterized in that, The in-vehicle interaction data includes: touch data, voice data, and physical operation data; the encoded sub-features include touch sub-features, voice sub-features, and operation sub-features. The data types based on the in-vehicle interaction data are encoded using corresponding encoding methods to obtain encoded sub-features corresponding to each data type, including: Based on the touch data, a touch sequence is determined, and the touch sequence is sequence encoded to obtain the touch sub-features; Extract the prosodic features of the speech data and obtain the semantic features of the corresponding text data. Then, fuse the semantic features and the prosodic features to obtain the speech sub-features. The physical operation data is subjected to feature statistics to obtain at least one statistical feature for each physical operation, and the statistical features of each physical operation are fused to obtain the operation sub-features.

7. The method according to claim 6, characterized in that, The touch data includes at least one of touch coordinates, touch pressure value, touch area, and touch click interval; the voice data includes at least one of voice command text, acoustic features, and tone energy; and the physical operation data includes at least one of the rotation angle of physical knobs, the pressing depth of physical buttons, and the change in grip force on the steering wheel.

8. A device for processing vehicle occupant data, characterized in that, The device includes: The in-vehicle feature acquisition module is used to acquire in-vehicle perception data corresponding to the target member in the vehicle, and to extract features from the in-vehicle perception data to obtain a first feature. The in-vehicle perception data includes at least one of the following: at least one in-vehicle interaction data generated by the target member interacting with the vehicle, and at least one of the physiological state data of the target member collected by a wearable device. The first feature includes an in-vehicle behavior feature vector corresponding to the at least one in-vehicle interaction data and a physiological state feature vector corresponding to the physiological state data. The channel interaction feature acquisition module is used to acquire the interaction data of the target member in at least one interaction channel, and to extract features from the interaction data to obtain a second feature. The interaction data includes at least one of the following: interaction log data obtained from the target website, interaction feedback data obtained from the target mini-program, and page jump path obtained from the target client. The second feature includes: a first feature vector corresponding to the target website, a second feature vector corresponding to the target mini-program, and a third feature vector corresponding to the target client. The feature fusion module is used to construct an input matrix based on the first feature and the second feature, wherein each row of the input matrix represents a feature vector; calculate the attention weight between any two rows of the input matrix based on the input matrix to obtain an attention weight matrix; multiply the input matrix and the attention weight matrix to obtain the state feature of the target member; and perform mapping processing on the state feature to obtain the member state of the target member. The channel interaction feature acquisition module is used to: respond to interaction data including interaction log data; generate a user behavior sequence based on the interaction log data and perform sequence encoding on the user behavior sequence to obtain a first feature vector; respond to interaction data including interaction feedback data; perform feature encoding on the interaction feedback data to obtain a second feature vector; respond to interaction data including page jump paths; preprocess the page jump path into a directed graph, wherein nodes in the directed graph represent business units in the target client, and edges in the directed graph represent jump relationships between two connected business units; encode the directed graph using a graph neural network to obtain a third feature vector; and determine the second feature based on at least one of the first feature vector, the second feature vector, and the third feature vector.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images