Data processing method, device and equipment based on vehicle-mounted system and medium

Abstract

The embodiment of the invention provides a data processing method based on a vehicle-mounted system, and the method is applied to the field of vehicle services, and comprises the steps: obtaining user image data collected by a vehicle image collection system, extracting local feature information and global feature information in the user image data, determining the similarity between the local feature information and the global feature information, fusing the local feature information and the global feature information according to the similarity to obtain feature fusion information, classifying the feature fusion information to obtain a classification result of the user image data, determining user attribute information according to the classification result, and determining the user attribute information according to the user attribute information. According to the method, the personalized information is determined, the personalized operation corresponding to the personalized information is executed, the attribute information of the passenger is accurately identified through fusion of the local features and all the features of the user image data, the personalized service is provided according to the attribute information of the passenger, and the user experience and the satisfaction degree are improved.

Description

Technical Field

[0001] The present application relates to the technical field of human-computer interaction, and in particular to a data processing method, device, equipment and medium based on a vehicle-mounted system. Background Art

[0002] With the development of intelligent vehicles, cockpit design is gradually changing from a simple means of transportation to a highly personalized mobile living space. The cockpit is no longer just a place for driving and riding, but a comprehensive experience environment that integrates multiple functions such as comfort, entertainment, and connectivity. This transformation not only improves the convenience and safety of driving, but also brings passengers an unprecedented personalized experience. Whether it is a long-distance trip or daily commuting, the smart cockpit can provide customized services according to the needs of passengers, making every trip a pleasure.

[0003] For example, older passengers tend to focus more on cabin comfort and convenience, while younger passengers tend to pursue rich entertainment systems and connectivity functions, hoping to enjoy music, videos and social interactions during their journey.

[0004] In related technologies, users often need to manually select their favorite information, such as music, videos, navigation routes, etc. Although this interactive method is simple and direct, it also has some limitations. Users need to spend time and energy to search and select information, and the operation is relatively cumbersome. Especially during driving, manual operation poses safety risks; it lacks personalization and cannot provide accurate services based on user preferences and habits. Summary of the invention

[0005] In view of the above problems, a data processing method, device, equipment and medium based on a vehicle-mounted system are proposed to overcome the above problems or at least partially solve the above problems, including:

[0006] A data processing method based on a vehicle-mounted system, the method comprising:

[0007] Acquire user image data collected by the vehicle image acquisition system;

[0008] Extracting local feature information and global feature information from the user image data;

[0009] Determining the similarity between the local feature information and the global feature information, and fusing the local feature information with the global feature information based on the similarity to obtain feature fusion information;

[0010] Classifying the feature fusion information to obtain a classification result of the user image data;

[0011] Determining user attribute information according to the classification result;

[0012] According to the user attribute information, personalized information is determined, and a personalized operation corresponding to the personalized information is performed.

[0013] In an optional embodiment of the present application, the user attributes include at least an age attribute and a gender attribute, and the determining of personalized information based on the user attribute information and performing a personalized operation corresponding to the personalized information includes:

[0014] Determine, based on the age attribute and the gender attribute, the recommended information and the corresponding attention amount of the recommended information within a preset time period;

[0015] The recommended information whose attention amount exceeds a preset attention threshold is determined as personalized information, and the personalized information is recommended to the user.

[0016] In an optional embodiment of the present application, determining personalized information based on the user attribute information and performing a personalized operation corresponding to the personalized information includes:

[0017] Determining personalized information according to the age attribute and the gender attribute, the personalized information including target driving mode information and target cockpit environment information;

[0018] adjusting the current driving mode of the vehicle according to the target driving mode information;

[0019] The current cockpit environment of the vehicle is adjusted according to the target cockpit environment information.

[0020] In an optional embodiment of the present application, the extracting the local feature information and the global feature information from the user image data includes:

[0021] Inputting the user image data into a trained first encoder to obtain local feature information of a first size, wherein the local feature information at least includes contour information of the user image data;

[0022] The user image data is input into a trained second encoder, and the resolution of the user image data is reduced to obtain global feature information of a second size.

[0023] In an optional embodiment of the present application, determining the similarity between the local feature information and the global feature information, and fusing the local feature information with the global feature information based on the similarity to obtain feature fusion information includes:

[0024] Adjusting the global feature information of the second size according to the first size to obtain the global feature information of the first size;

[0025] Splicing the local feature information of the first size with the global feature information of the first size to obtain a splicing result;

[0026] Calculating a similarity coefficient matrix corresponding to the splicing result, wherein the similarity coefficient matrix represents the similarity between the local feature information of the first size and the global feature information of the first size;

[0027] Multiplying the local feature information by the similarity coefficient matrix to obtain a first target result;

[0028] Multiplying the global feature information by the similarity coefficient matrix to obtain a second target result;

[0029] The first target result and the second target result are added to obtain feature fusion information.

[0030] In an optional embodiment of the present application, classifying the feature fusion information to obtain a classification result of the user image data includes:

[0031] Using a preset semantic classification component to classify the feature fusion information to obtain a type corresponding to each pixel in the user image data;

[0032] A classification result of the user image data is determined according to the type corresponding to each pixel in the user image data.

[0033] In an optional embodiment of the present application, before acquiring the user image data acquired by the vehicle image acquisition system, the process includes:

[0034] Annotating the user image training data by using a preset large segmentation model to obtain a simulated reference label;

[0035] Checking whether the simulated reference label matches the real reference label corresponding to the user image training data;

[0036] In a case where the simulated reference label matches the real reference label corresponding to the user image training data, using the user image training data and the simulated reference label as training samples;

[0037] The training samples are used to train the preset first encoder and the preset second encoder respectively to obtain the trained first encoder and the trained second encoder.

[0038] A data processing device based on a vehicle-mounted system, the device comprising:

[0039] An image acquisition module, configured to acquire user image data acquired by a vehicle image acquisition system;

[0040] A feature extraction module, configured to extract local feature information and global feature information from the user image data;

[0041] a feature fusion module, configured to determine the similarity between the local feature information and the global feature information, and fuse the local feature information with the global feature information according to the similarity to obtain feature fusion information;

[0042] An image classification module is configured to classify the feature fusion information to obtain a classification result of the user image data;

[0043] an attribute determination module, configured to determine user attribute information according to the classification result;

[0044] The operation execution module is configured to determine personalized information according to the user attribute information, and execute the personalized operation corresponding to the personalized information.

[0045] A vehicle comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the data processing method based on the vehicle-mounted system as described above is implemented.

[0046] A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the data processing method based on the vehicle-mounted system as described above is implemented.

[0047] The embodiments of the present application have the following advantages:

[0048] By acquiring user image data collected by a vehicle image acquisition system, extracting local feature information and global feature information in the user image data, determining the similarity between the local feature information and the global feature information, and based on the similarity, fusing the local feature information with the global feature information to obtain feature fusion information, classifying the feature fusion information to obtain a classification result of the user image data, determining user attribute information based on the classification result, determining personalized information based on the user attribute information, and performing personalized operations corresponding to the personalized information, the present application accurately identifies the attribute information of passengers by fusing local features and all features of user image data, and provides personalized services based on the attribute information of passengers, thereby improving user experience and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 is a flowchart of a data processing method based on a vehicle-mounted system provided in some embodiments of the present application;

[0051] Figure 2 is a flowchart of another method for processing data based on an in-vehicle system provided by some embodiments of the present application;

[0052] Figure 3 is an image processing model architecture diagram provided by some embodiments of the present application;

[0053] Figure 4 is a flowchart of another method for processing data based on a vehicle-mounted system provided by an embodiment of the present application;

[0054] Figure 5 is a flowchart of another method for processing data based on a vehicle-mounted system provided by an embodiment of the present application;

[0055] Figure 6 is a general flow chart of data processing provided by some embodiments of the present application;

[0056] Figure 7 It is a structural diagram of a data processing device based on a vehicle-mounted system provided in some embodiments of the present application. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0058] Reference Figure 1 , shows a flowchart of a data processing method based on a vehicle-mounted system provided in some embodiments of the present application, which may specifically include the following steps:

[0059] Step 101: Acquire user image data collected by a vehicle image collection system;

[0060] Among them, the vehicle image acquisition system refers to the equipment and systems installed inside or outside the vehicle, which are used to collect image data of the vehicle's surrounding environment or passengers in the vehicle. These systems usually include cameras, image sensors, image processing units and related software algorithms for real-time capture, processing and analysis of image information. User image data refers to image information related to the user obtained through the image acquisition system. This data may include the user's facial image, body posture, gestures, expressions, etc., which are used to analyze the user's identity, status, behavior and other information.

[0061] In some implementations of this embodiment, the vehicle image acquisition system may be an OMS (Occupant Monitoring System), which is an intelligent system installed in the vehicle and is intended to monitor and evaluate the status and behavior of passengers in the vehicle in real time. The system usually combines a variety of sensors, cameras and advanced algorithms to ensure the safety and comfort of passengers. The vehicle image acquisition system may also be a DMS (Driver Monitoring System) system, which may be an advanced technology for monitoring the status and behavior of drivers. The system is intended to improve driving safety and prevent traffic accidents caused by driver fatigue, distraction or improper behavior. The DMS system usually uses cameras, sensors and algorithms to monitor the physiological and behavioral characteristics of drivers in real time. Specifically, the user's facial image, body posture, etc. can be captured by the DMS system or the OMS system, for example, an infrared camera is used to capture infrared image data of the user in a low-light or no-light environment, and a depth camera is used to capture the user's depth information.

[0062] Step 102: extracting local feature information and global feature information from the user image data.

[0063] Among them, local feature information refers to the features of a specific area in the image, usually local details or local patterns in the image. These features usually have high invariance and can resist image transformations such as translation, rotation, and scaling. Global feature information refers to the features that describe the entire image or a larger area in the image, usually reflecting the overall structure, color distribution, texture and other global properties of the image.

[0064] In some implementations of this embodiment, local feature information and global feature information in user image data may be extracted using a pre-trained model.

[0065] Step 103: determining the similarity between the local feature information and the global feature information, and fusing the local feature information and the global feature information based on the similarity to obtain feature fusion information;

[0066] In the above steps, local feature information and global feature information in the user image data are extracted through a pre-trained model. In some embodiments of the present application, the pre-trained model used to extract local feature information and global feature information can be two different models. Different models may have obvious differences in feature extraction and calculation mechanisms. It is crucial to utilize the complementary advantages of local feature information and global feature information to improve the expressiveness of the extracted features.

[0067] In some implementations of this embodiment, it is assumed that T and F represent intermediate features from two different models, respectively. First, feature T can be reshaped to match the same shape of feature F, and then the similarity between the local feature information T and the global feature information F is determined, and based on the similarity, the local feature information and the global feature information are fused to obtain feature fusion information. Among them, the similarity can be specifically a correlation coefficient matrix, assuming that the correlation coefficient matrix is ​​M, and then the matrix is ​​multiplied with the original features T and F respectively to obtain F' and T', and the two are added to obtain the final output Z. The embodiment of the present application combines the advantages of local features and global features to obtain more comprehensive and robust feature fusion information.

[0068] Step 104: classify the feature fusion information to obtain a classification result of the user image data;

[0069] Feature fusion information is a comprehensive feature representation obtained by combining local feature information and global feature information. In order to further analyze and understand these features, it is usually necessary to classify them to obtain the classification results of user image data.

[0070] In some implementations of this embodiment, a semantic segmentation head can be used to classify feature fusion information to obtain a classification result of the user image, wherein the semantic segmentation head is the part of the deep learning model used to perform semantic segmentation tasks. The semantic segmentation task refers to a computer vision task, the goal of which is to assign each pixel in the image to a predefined category, thereby generating a pixel-level classification of the image. In the present application, the output of the semantic segmentation head can be a semantic segmentation map that is consistent with the width and height of the input image, that is, after the user image data is subjected to feature extraction, feature fusion and classification, the classification result obtained can be a semantic segmentation map, which is represented in the form of an image, in which the color or grayscale value of each pixel represents the category to which it belongs.

[0071] Step 105: Determine user attribute information based on the classification result;

[0072] In some implementations of this embodiment, the above-mentioned pre-trained model can be quantized by QNN (Quantized Neural Network) and deployed on the vehicle side, wherein QNN quantization refers to the use of low-precision (such as 8-bit, 4-bit or even lower) numerical representation in the neural network to replace the traditional 32-bit floating point representation, thereby reducing computational complexity and memory usage, and improving model reasoning speed and energy efficiency. The quantized model is used to extract features from the user image data collected by the image acquisition system to obtain local feature information and global feature information, which are fused, and the fused information is classified to obtain the classification result of the user image data, and the user attributes are determined based on the classification result.

[0073] Step 106: Determine personalized information based on the user attribute information, and perform personalized operations corresponding to the personalized information.

[0074] In some implementations of this embodiment, the age and gender of pedestrians in the cockpit scene can be inferred and determined, and then these attributes can be used to provide personalized services to them, such as determining some personalized information and performing personalized operations corresponding to the personalized information. For example, according to the classification results, it is determined that the user attribute information is a boy born in the 1990s, and the in-vehicle system actively recommends an R&B style song to him.

[0075] The embodiment of the present application obtains user image data collected by a vehicle image acquisition system, extracts local feature information and global feature information in the user image data, determines the similarity between the local feature information and the global feature information, and fuses the local feature information with the global feature information based on the similarity to obtain feature fusion information, classifies the feature fusion information to obtain a classification result of the user image data, determines user attribute information based on the classification result, determines personalized information based on the user attribute information, and performs personalized operations corresponding to the personalized information. The present application accurately identifies the attribute information of passengers by fusing local features and all features of user image data, and provides personalized services based on the attribute information of passengers, thereby improving user experience and satisfaction.

[0076] Reference Figure 2 , shows a flowchart of another method for processing data based on a vehicle-mounted system provided by an embodiment of the present application, which may specifically include the following steps:

[0077] Step 201: Acquire user image data collected by a vehicle image collection system.

[0078] In a specific implementation, the user image data may be infrared passenger image data, which is passenger image data collected by an infrared camera. Infrared light is an electromagnetic wave with a longer wavelength and can capture images in low-light or no-light environments. Infrared light imaging does not rely on visible light and can capture images without infringing on privacy. Infrared light can capture the thermal radiation of objects and generate thermal imaging images. Therefore, the user image data collected in the embodiments of the present application has the advantages of low-light environment imaging, thermal imaging, non-contact detection and privacy protection.

[0079] In some embodiments of the present application, the following steps may be included before step 201:

[0080] Sub-step 11: Label the user image training data using the preset large segmentation model to obtain simulated reference labels.

[0081] Sub-step 12: Check whether the simulated reference label matches the real reference label corresponding to the user image training data;

[0082] Sub-step 13: When the simulated reference label matches the real reference label corresponding to the user image training data, use the user image training data and the simulated reference label as training samples;

[0083] Sub-step 14: using the training samples to train the preset first encoder and the preset second encoder respectively, to obtain the trained first encoder and the trained second encoder.

[0084] Specifically, the preset large segmentation model may be a SAM (Segment Anything Model) with zero-sample generalization capability. The SAM with zero-sample generalization capability has achieved good results in task execution in general scenarios, but its segmentation performance under infrared light data in cockpit scenarios needs to be improved.

[0085] In order to improve the labeling efficiency, a small amount of task data can be manually labeled first. The manually labeled labels are called true reference labels.

[0086] The user image training data is labeled by a preset SAM model to obtain a simulated reference label. By checking whether the simulated reference label matches the manually labeled real reference label, it is determined whether the SAM model needs to be adjusted. When the simulated reference label matches the real reference label corresponding to the user image training data, the user image training data and the simulated reference label labeled by the SAM model are used as training samples. If the simulated reference label does not match the real reference label, the SAM model needs to be fine-tuned to obtain an adjusted SAM model, and then the user image training data is labeled again by the adjusted SAM model to obtain a simulated reference label, and when the simulated reference label matches the real reference label, the user image training data and the simulated reference label labeled by the SAM model are used as training samples.

[0087] Step 202: extracting local feature information and global feature information from the user image data.

[0088] In some embodiments of the present application, step 202 may include the following sub-steps:

[0089] Sub-step 21: input the user image data into a trained first encoder to obtain local feature information of a first size, where the local feature information at least includes contour information of the user image data.

[0090] Sub-step 22: input the user image data into a trained second encoder, and reduce the resolution of the user image data to obtain global feature information of a second size.

[0091] like Figure 3 As shown, it is an architecture diagram of an image processing model provided in some embodiments of the present application. The architecture mainly includes three components: a trained first encoder with hierarchical perceptual pixel activation enhancement, a trained second encoder, and a lightweight decoder.

[0092] Among them, the trained first encoder may refer to a CNN encoder with hierarchical perceptual pixel activation enhancement, and the trained second encoder may refer to a Transformer encoder. The first encoder CNN is used to extract local feature information in user image data, and the second encoder Transformer is used to extract global features of user image data.

[0093] In the training stage of the first encoder and the second encoder, the user image training data determined by the SAM model in sub-step 14 can be used as training samples, these training sample image data (i.e., pictures) are scaled and adjusted in equal proportions, and random noise, ColorJitter operation and normalization are added, and then the processed data is input into the Figure 3In the architecture, the first encoder, the second encoder and the classifier are trained to obtain the trained first encoder and the trained second encoder, the trained model is quantized by QNN, and deployed to the vehicle end equipped with Qualcomm chip. Among them, adding random noise refers to introducing randomly generated noise into the image data to simulate the noise and interference in the real world. The ColorJitter operation is a data enhancement technology used to randomly change the color attributes of the image during the image processing process. Normalization processing refers to scaling the image data to a specific range, which can be [0, 1] or [-1, 1], to facilitate model training and optimization. This application generates more diverse training data by introducing noise, thereby improving the generalization ability of the model, so that the model can still maintain good performance in the face of noise and interference; adjusts the brightness, contrast, saturation and hue of the image through the ColorJitter operation to generate diverse training data, thereby improving the generalization ability and robustness of the model; scales the image data to a specific range through normalization processing, accelerates model convergence, and improves training stability.

[0094] In the specific implementation, the user image data input to the trained first encoder and the second encoder refers to the three-channel cockpit infrared user data with a size of 480*288, but is not limited to the three-channel cockpit infrared user data with a size of 480*288. The CNN encoder is used to generate a feature map F with a size of 60*36*Cf. At the same time, on another path, the user image data enters the Efficient Transformer encoder after being processed by the Transformer stem block to obtain the feature embedding T∈R N×D ,in Indicates the number of markers, D = C t ×P 2 represents the dimension of each marker and P represents the marker size.

[0095] In the specific implementation, you can also Figure 3 In the first encoder stem block (i.e., CNN stem block) in , three 3×3 convolutional layers are used, and the last layer has a stride of 2, which can be obtained as The feature map of the cockpit infrared user data is three-channel with a size of 480*288. Figure 3 The output of the first encoder stem block in is a feature map of 240×144×C1, where C1 represents the number of output channels. In the embodiment of the present application, four 3×3 convolutional layers are used in the Transformer stem block with a step size of 2, resulting in an output feature size of That is, the Transformer stem block in the Transformer encoder reduces the resolution while extracting feature representations, contributing to the lightweight design of the model by minimizing the computational load.

[0096] First, the specific structure and data processing process of the first encoder CNN encoder in this embodiment are described. Figure 3 It can be seen that the output of the CNN encoder stem block in this embodiment will also be processed by the hierarchical perception pixel excitation module to improve the extraction of pixel features. Before inputting to the hierarchical perception pixel excitation module, it is also necessary to perform a downsampling operation through a downsampling operation module. The downsampling operation can be maximum pooling, average pooling or stride convolution. The purpose of the downsampling operation is to reduce the height and width of the feature map, thereby reducing the amount of calculation of subsequent layers.

[0097] The hierarchical perception pixel excitation module uses convolutions of different kernel sizes in the same layer, combined with pixel activation, to extract features from objects of different sizes, and uses a multi-scale strategy to capture unique pixel features at different receptive field levels. This module avoids redundant calculations, while maintaining feature validity, making the network more streamlined. This module enhances the model's ability to effectively identify objects of various sizes in an image, ultimately reducing the pixel misclassification rate. Specifically, for a given feature input in the hierarchical perception pixel excitation module, First, it is input into a convolutional layer to reduce its channel dimension to That is, the output feature map is expressed as f 1×1 Represents a convolution operation with a kernel size of 1×1. This dimensionality reduction facilitates channel operations in subsequent hierarchical convolutional layers. Subsequently, four parallel convolution operations are performed, including decomposed convolution and depth-wise separable convolution. The kernel sizes of the four parallel convolution operations are 3, 3, 5, and 7, respectively, which is what is referred to as convolutions of different kernel sizes in the same layer. In order to further enhance the receptive field, the last three convolution layers of the four parallel convolution operations of this application use dilated convolutions. This strategy enables the model to capture image features across various scales, ensuring comprehensive and detailed information extraction.

[0098] The above process can be expressed as:

[0099]

[0100] Among them, l1 represents the intermediate feature, f 1×3 Represents a convolution operation with a kernel size of 3.

[0101] Figure 4 A schematic diagram of a hierarchical perception pixel activation module provided in some embodiments of the present application, such as Figure 4 As shown in , after performing four parallel convolution operations in the hierarchical perception pixel activation module, a pixel excitation module can be used to enhance the expressiveness of the features. Specifically, the pixel excitation module can be a content-aware spatial attention mechanism. In the pixel excitation module, the outputs of the four parallel convolution operations are first sent to the global average pooling layer to generate the corresponding feature map, and then the generated feature map is reshaped and flattened. The reshape operation refers to changing the shape or dimension of the data without changing the data content to meet the needs of operations at different layers; the flattening operation refers to converting multi-dimensional data into one-dimensional data. After the reshape and flattening operations, it is input into the Softmax function to calculate the weight matrix. The Softmax function is an activation function commonly used in multi-classification problems. It converts a real vector into a probability distribution. Specifically, the Softmax function converts each element of the input vector to a value between 0 and 1, and the sum of all output values ​​is 1. This application calculates the weight matrix through the Softmax function, and then multiplies the weight matrix with the input features to obtain a content-aware attention enhancement output, which is the input of the pixel excitation module. The input of the pixel excitation module is added and a 1×1 convolution operation is performed again to obtain a preliminary result. This preliminary result is added to the initial input of the hierarchical perception pixel activation module, which is the output of a hierarchical perception pixel activation module.

[0102] Next, the Transformer encoder in the architecture of this embodiment is explained. When processing high-resolution inputs, the real-time model of the traditional Transformer encoder is too large. Due to the limited resources on the vehicle side, the traditional Transformer encoder cannot be applied on the vehicle side. The embodiment of the present application reduces the computational cost of the Transformer encoder by reducing the feature dimension without significantly losing image details. This embodiment introduces a spatial reduction linear projection method, which first maps the features to a latent embedding space with a low dimension, and then uses it for multi-head self-attention calculations. This method is called efficient multi-head self-attention with learned projection and segmentation operations.

[0103] In efficient multi-head self-attention, the input feature is represented as Among them C t , H t and W t Represents the number of channels, height and width in the feature map respectively. After the Reshape operation, a series of flat non-overlapping patches are obtained. The result is in Represents the number of patches (i.e., the length of the input sequence), and each patch is of size P×P. Subsequently, these patches are mapped to a potential D-dimensional embedding space through a learnable linear projection layer, denoted as Z∈R N×D This process can be expressed as

[0104]

[0105] in, represents the i-th Patch. Note that the position embedding is omitted here to allow greater adaptability to different input sizes.

[0106] Then, the three matrices of query Q, key K and value V in Transformer can be obtained by their linear projection It can be expressed as

[0107]

[0108] In addition, the number of heads h in the multi-head self-attention is also a user-defined parameter, ensuring that the dimension of each head is equal to The dimensions of q, k, and v in the i-th head are N×d. In the i-th head, k and v are spatially reduced by a factor r, where r is the reduction ratio, set to 2. Then, the sub-tokens produced by the feature splitting operation are matrix multiplied with a field that only represents the original perception 1 / s, where s represents the number of feature splits, set to 4. This process can be described as

[0109] [q1,…,q s ],[k1,…,k s ],[v1,…,v s ]=Feature_Split(q,k,v)

[0110] This process is similar to the concept of group convolution and can effectively reduce memory consumption. Therefore, the Transformer encoder in the embodiment of the present application occupies a small space and can be deployed on the vehicle side with limited resources.

[0111] Step 203: Determine the similarity between the local feature information and the global feature information, and fuse the local feature information and the global feature information based on the similarity to obtain feature fusion information.

[0112] In some embodiments of the present application, step 203 includes the following sub-steps:

[0113] Sub-step 31: adjusting the global feature information of the second size according to the first size to obtain the global feature information of the first size;

[0114] Sub-step 32: splicing the local feature information of the first size with the global feature information of the first size to obtain a splicing result;

[0115] Sub-step 33: calculating a similarity coefficient matrix corresponding to the splicing result, where the similarity coefficient matrix represents the similarity between the local feature information of the first size and the global feature information of the first size;

[0116] Sub-step 34: multiplying the local feature information with the similarity coefficient matrix to obtain a first target result;

[0117] Sub-step 35: multiplying the global feature information and the similarity coefficient matrix to obtain a second target result;

[0118] Sub-step 36: Add the first target result and the second target result to obtain feature fusion information.

[0119] In some implementations of this embodiment, the two different types of local feature information F and global feature information T can be coordinated through a correlation weighted fusion module, wherein the correlation weighted fusion module is Figure 3 The fusion operation in , through the fusion operation, fuses the CNN features (i.e., local feature information F) and Transformer features (i.e., global feature information T) with high correlation. This fusion of related CNN and Transformer features enhances the recovery of boundary information and lightweight decoder segmentation head.

[0120] In the first step, the global feature information of the second size is adjusted according to the first size to obtain the global feature information of the first size. Specifically, the Transformer feature T can be reshaped to match the same shape of the CNN feature F. In the second step, the local feature information of the first size is spliced ​​with the global feature information of the first size to obtain a splicing result. Specifically, the two feature sets of the local feature information and the adjusted global feature information can be post-spliced. The third step is to calculate the similarity coefficient matrix corresponding to the splicing result, where the similarity coefficient matrix represents the similarity between the local feature information of the first size and the global feature information of the first size. Specifically, in order to further reduce the computational cost, the depth-separable convolution can be used to reduce the channel dimension. After the global average pooling (GAP) and Sigmoid operations, the correlation coefficient matrix M is calculated, where the global average pooling is a pooling operation that performs a global average on each channel of the input feature map to obtain a single value. Specifically, for a feature map of shape (C, H, W), the output of the GAP operation is a tensor of shape (C, 1, 1), that is, the average value of each channel; the Sigmoid function is a nonlinear activation function that maps the input value to a range between 0 and 1; the correlation coefficient matrix M is used to measure the correlation or dependency between different categories. The correlation coefficient matrix M is a C×C matrix, where each element M ij Represents the correlation coefficient between the i-th category and the j-th category. The fourth step is to multiply the local feature information with the similarity coefficient matrix to obtain the first target result, multiply the global feature information with the similarity coefficient matrix to obtain the second target result, and add the first target result to the second target result to obtain feature fusion information. Specifically, the correlation coefficient matrix M is then multiplied with the original features F and T to obtain the corresponding first target result F' and second target result T', and the two are added to obtain the final output Z as the feature fusion information.

[0121] Step 204: Classify the feature fusion information to obtain a classification result of the user image data.

[0122] In some embodiments of the present application, step 204 may include the following sub-steps:

[0123] Sub-step 41: using a preset semantic classification component to classify the feature fusion information, and obtaining the type corresponding to each pixel in the user image data.

[0124] Sub-step 42: Determine the classification result of the user image data according to the type corresponding to each pixel in the user image data.

[0125] Among them, the preset semantic classification components are Figure 3The semantic segmenter or classifier in the image is used to classify each pixel in the image to determine the category to which it belongs. Semantic segmentation is an important task in computer vision, which aims to assign each pixel in the image to a predefined category, thereby achieving pixel-level classification of the image. In the specific implementation, the semantic classification component is used to output a semantic segmentation map that is consistent with the width and height of the input image. The number of channels of the map is consistent with the number of semantic categories. Semantic categories refer to different object or region categories that need to be identified and segmented in the image. Each semantic category represents a specific object or region in the image, such as "hair", "facial features", "limbs", etc. The number of semantic categories refers to the total number of different semantic categories that need to be identified and segmented in the semantic segmentation task. This number determines the number of channels of the semantic segmentation map. Assuming that the width and height of the input image are H×W, and the number of semantic categories is 3, the dimension of the output semantic segmentation map is H×W×3. Each channel corresponds to a semantic category. For example, channel 1 represents the segmentation result of the "hair" category, channel 2 represents the segmentation result of the "facial features" category, and channel 3 represents the segmentation result of the "limbs" category.

[0126] Step 205: Determine user attribute information based on the classification result.

[0127] The classification result of the user image data obtained through the above steps is used to determine the attribute information of the user. The user attributes include at least age attribute and gender attribute.

[0128] Step 206: Determine the recommended information and the corresponding attention amount of the recommended information within a preset time period according to the age attribute and the gender attribute;

[0129] After determining the age and gender attributes of the users in the cockpit, information can be recommended to the users based on their age and gender attributes, and the recommended information can be music, video, news, etc. In a specific implementation, the attention volume of the recommended information within a preset time period can also be determined, where the preset time period can be 1 day, 7 days or a month, and the preset time period can be adjusted according to actual conditions.

[0130] Step 207: Determine the recommended information whose attention amount exceeds a preset attention threshold as personalized information, and recommend the personalized information to the user.

[0131] Assuming that the preset time period is 1 day, news, videos or music are filtered based on the user's age and gender attributes, and the popularity of the recommended information is further filtered. For example, the most popular news, videos or music in a day are recommended to users to improve the user experience in the cabin.

[0132] The embodiment of the present application obtains the user image data collected by the vehicle image acquisition system, extracts the local feature information and the global feature information in the user image data, determines the similarity between the local feature information and the global feature information, and fuses the local feature information and the global feature information based on the similarity to obtain the feature fusion information, classifies the feature fusion information, obtains the classification result of the user image data, determines the user attribute information based on the classification result, determines the personalized information based on the user attribute information, and performs the personalized operation corresponding to the personalized information. The present application accurately identifies the attribute information of the passenger by fusing the local features and all features of the user image data, and provides personalized services based on the attribute information of the passenger, thereby improving the user experience and satisfaction. Among them, the user attribute information includes age information and gender information, and based on the age information and gender information, provides the user with recommended information whose attention amount exceeds the preset attention threshold within a preset time period, provides the user with rich hot information, and further meets the personalized needs of the user.

[0133] Reference Figure 5 , shows a flowchart of another method for processing data based on a vehicle-mounted system provided by an embodiment of the present application, which may specifically include the following steps:

[0134] Step 501: Acquire user image data collected by the vehicle image collection system;

[0135] Step 502: extracting local feature information and global feature information from the user image data;

[0136] Step 503: determining the similarity between the local feature information and the global feature information, and fusing the local feature information and the global feature information according to the similarity to obtain feature fusion information;

[0137] Step 504: classify the feature fusion information to obtain a classification result of the user image data;

[0138] Step 505: Determine user attribute information based on the classification result;

[0139] The execution method of step 501 to step 505 is the same as step 101 to step 105, so it will not be repeated. The user attribute information at least includes age attribute and gender attribute.

[0140] Step 506: determining personalized information according to the age attribute and the gender attribute, where the personalized information includes target driving mode information and target cockpit environment information;

[0141] Age attribute refers to the age range or specific age value of the people in the cabin. Refers to the gender of the people in the cabin, usually divided into male or female. The user's age attribute can be specifically divided into children, youth, middle-aged and elderly, and the user's gender attribute can be divided into male and female. Personalized information refers to information or settings customized according to the user's specific attributes (such as age, gender, etc.) to meet the user's personalized needs. Personalized information may include target driving mode information and target cabin environment information. Target driving mode information refers to the driving mode settings recommended for the driver based on the driver's age and gender attributes. These settings may include acceleration response, steering sensitivity, suspension hardness, etc. For young men, a sports driving mode may be recommended; for older women, a comfortable driving mode may be recommended. Target cabin environment information refers to the cabin environment settings recommended for the driver based on the driver's age and gender attributes. These settings may include seat position, temperature, ambient light color, audio system, etc. For example, for young men, dynamic red ambient lights and powerful sound effects may be recommended; for young women, soft pink ambient lights and soothing music may be recommended.

[0142] Step 507: adjusting the current driving mode of the vehicle according to the target driving mode information;

[0143] Step 508: Adjust the current cabin environment of the vehicle according to the target cabin environment information.

[0144] In some implementations of the embodiments of the present application, after determining the user attribute information and determining the target driving mode information and the target cabin environment information based on the user attribute information, the vehicle's current driving mode can be adjusted based on the target driving mode information, and the vehicle's current cabin environment can be adjusted based on the target cabin environment information. For example, the current driving mode can be adjusted to a sports driving mode; for another example, the current cabin environment can be adjusted to soft pink lights and soothing music can be played.

[0145] In order to describe this application as a whole, Figure 6 The figure shows the overall data processing flow chart provided by some embodiments of the present application, the first step is data input, the input data is a large amount of infrared passenger data collected by the vehicle-mounted OMS or DMS camera; the second step is data pre-labeling, which is to pre-label a large amount of infrared passenger data; the third step is data processing, which may include adding random noise, ColorJitter operation and normalization processing; the fourth step is model training, which is to train the model using data that has been pre-labeled and processed; the fifth step is model deployment, which is to deploy the lightweight model on the vehicle side; the sixth step is personalized service, which is to obtain the user's attribute information based on the recognition information of the model, and provide personalized services to the user based on the user's attribute information.

[0146] The embodiment of the present application obtains the user image data collected by the vehicle image acquisition system, extracts the local feature information and the global feature information in the user image data, determines the similarity between the local feature information and the global feature information, and fuses the local feature information and the global feature information based on the similarity to obtain the feature fusion information, classifies the feature fusion information, obtains the classification result of the user image data, determines the user attribute information based on the classification result, determines the personalized information based on the user attribute information, and performs the personalized operation corresponding to the personalized information. The present application accurately identifies the attribute information of the passenger by fusing the local features and all features of the user image data, and provides personalized services based on the attribute information of the passenger, thereby improving the user experience and satisfaction. Among them, the user attribute information includes age information and gender information, and based on the age information and gender information, provides the user with personalized content including the target driving information and the target cabin environment information, and adjusts the current driving model and the current cabin environment of the vehicle to further meet the personalized needs of the user.

[0147] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0148] Reference Figure 7 , shows a schematic diagram of the structure of a data processing device based on a vehicle-mounted system provided by an embodiment of the present application, which may specifically include the following modules:

[0149] The image acquisition module 701 is configured to acquire user image data acquired by the vehicle image acquisition system;

[0150] A feature extraction module 702 is configured to extract local feature information and global feature information from the user image data;

[0151] The feature fusion module 703 is configured to determine the similarity between the local feature information and the global feature information, and fuse the local feature information with the global feature information according to the similarity to obtain feature fusion information;

[0152] An image classification module 704 is configured to classify the feature fusion information to obtain a classification result of the user image data;

[0153] The attribute determination module 705 is configured to determine user attribute information according to the classification result;

[0154] The operation execution module 706 is configured to determine personalized information according to the user attribute information, and execute the personalized operation corresponding to the personalized information.

[0155] In an optional embodiment of the present application, the user attributes include at least an age attribute and a gender attribute, and the operation execution module 706 includes:

[0156] A first information determination submodule is configured to determine the recommended information and the corresponding attention amount of the recommended information within a preset time period according to the age attribute and the gender attribute;

[0157] The first information recommendation submodule is configured to determine the recommended information whose attention amount exceeds a preset attention threshold as personalized information, and recommend the personalized information to the user.

[0158] In an optional embodiment of the present application, the operation execution module 706 includes:

[0159] A second information determination submodule is configured to determine personalized information according to the age attribute and the gender attribute, wherein the personalized information includes target driving mode information and target cockpit environment information;

[0160] a second adjustment submodule, configured to adjust the current driving mode of the vehicle according to the target driving mode information;

[0161] The third adjustment submodule is configured to adjust the current cockpit environment of the vehicle according to the target cockpit environment information.

[0162] In an optional embodiment of the present application, the feature extraction module 702 includes:

[0163] A local information extraction submodule is configured to input the user image data into a trained first encoder to obtain local feature information of a first size, wherein the local feature information at least includes contour information of the user image data;

[0164] The global information extraction submodule is configured to input the user image data into a trained second encoder and reduce the resolution of the user image data to obtain global feature information of a second size.

[0165] In an optional embodiment of the present application, the feature fusion module 703 includes:

[0166] a first adjustment submodule, configured to adjust the global feature information of the second size according to the first size to obtain the global feature information of the first size;

[0167] A first splicing submodule is configured to splice the local feature information of the first size with the global feature information of the first size to obtain a splicing result;

[0168] a similarity calculation submodule, configured to calculate a similarity coefficient matrix corresponding to the splicing result, wherein the similarity coefficient matrix represents the similarity between the local feature information of the first size and the global feature information of the first size;

[0169] A first multiplication submodule is configured to multiply the local feature information with the similarity coefficient matrix to obtain a first target result;

[0170] A second multiplication submodule is configured to multiply the global feature information by the similarity coefficient matrix to obtain a second target result;

[0171] The first fusion submodule is configured to add the first target result and the second target result to obtain feature fusion information.

[0172] In an optional embodiment of the present application, the image classification module 704 includes:

[0173] A first classification submodule is configured to classify the feature fusion information using a preset semantic classification component to obtain a type corresponding to each pixel in the user image data;

[0174] The second classification submodule is configured to determine a classification result of the user image data according to a type corresponding to each pixel in the user image data.

[0175] In an optional embodiment of the present application, the device further includes:

[0176] A first labeling module is configured to label the user image training data by using a preset large segmentation model to obtain a simulated reference label;

[0177] A first checking module is configured to check whether the simulated reference label matches the real reference label corresponding to the user image training data;

[0178] a training data determination module configured to use the user image training data and the simulated reference label as training samples if the simulated reference label matches the real reference label corresponding to the user image training data;

[0179] The first training module is configured to use the training samples to train the preset first encoder and the preset second encoder respectively to obtain the trained first encoder and the trained second encoder.

[0180] An embodiment of the present application also provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the above-mentioned data processing method based on the vehicle-mounted system is implemented.

[0181] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data processing method based on the vehicle-mounted system is implemented.

[0182] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0184] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0185] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0186] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0189] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0190] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0191] The above is a detailed introduction to the provided vehicle-mounted system-based data processing method, device, equipment and medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method based on a vehicle-mounted system, characterized in that: The method comprises: Acquire user image data collected by the vehicle image acquisition system; Extracting local feature information and global feature information from the user image data; Determining the similarity between the local feature information and the global feature information, and fusing the local feature information with the global feature information based on the similarity to obtain feature fusion information; Classifying the feature fusion information to obtain a classification result of the user image data; Determining user attribute information according to the classification result; According to the user attribute information, personalized information is determined, and a personalized operation corresponding to the personalized information is performed.

2. The method according to claim 1, characterized in that The user attributes include at least an age attribute and a gender attribute, and the determining of personalized information based on the user attribute information and performing a personalized operation corresponding to the personalized information includes: Determine, based on the age attribute and the gender attribute, the recommended information and the corresponding attention amount of the recommended information within a preset time period; The recommended information whose attention amount exceeds a preset attention threshold is determined as personalized information, and the personalized information is recommended to the user.

3. The method according to claim 2, characterized in that The determining of personalized information based on the user attribute information and performing a personalized operation corresponding to the personalized information includes: Determining personalized information according to the age attribute and the gender attribute, the personalized information including target driving mode information and target cockpit environment information; adjusting the current driving mode of the vehicle according to the target driving mode information; The current cockpit environment of the vehicle is adjusted according to the target cockpit environment information.

4. The method according to claim 1, characterized in that: The extracting of local feature information and global feature information from the user image data includes: Inputting the user image data into a trained first encoder to obtain local feature information of a first size, wherein the local feature information at least includes contour information of the user image data; The user image data is input into a trained second encoder, and the resolution of the user image data is reduced to obtain global feature information of a second size.

5. The method according to claim 4, characterized in that The determining the similarity between the local feature information and the global feature information, and fusing the local feature information with the global feature information according to the similarity to obtain feature fusion information, includes: Adjusting the global feature information of the second size according to the first size to obtain the global feature information of the first size; Splicing the local feature information of the first size with the global feature information of the first size to obtain a splicing result; Calculating a similarity coefficient matrix corresponding to the splicing result, wherein the similarity coefficient matrix represents the similarity between the local feature information of the first size and the global feature information of the first size; Multiplying the local feature information by the similarity coefficient matrix to obtain a first target result; Multiplying the global feature information by the similarity coefficient matrix to obtain a second target result; The first target result and the second target result are added to obtain feature fusion information.

6. The method according to any one of claims 1 to 5, characterized in that: The classifying the feature fusion information to obtain the classification result of the user image data includes: Using a preset semantic classification component to classify the feature fusion information to obtain a type corresponding to each pixel in the user image data; A classification result of the user image data is determined according to the type corresponding to each pixel in the user image data.

7. The method according to claim 4, characterized in that Before obtaining the user image data collected by the vehicle image collection system, the method includes: Annotating the user image training data by using a preset large segmentation model to obtain a simulated reference label; Checking whether the simulated reference label matches the real reference label corresponding to the user image training data; In a case where the simulated reference label matches the real reference label corresponding to the user image training data, using the user image training data and the simulated reference label as training samples; The training samples are used to train the preset first encoder and the preset second encoder respectively to obtain the trained first encoder and the trained second encoder.

8. A data processing device based on a vehicle-mounted system, characterized in that: The device comprises: An image acquisition module, configured to acquire user image data acquired by a vehicle image acquisition system; A feature extraction module, configured to extract local feature information and global feature information from the user image data; a feature fusion module, configured to determine the similarity between the local feature information and the global feature information, and fuse the local feature information with the global feature information according to the similarity to obtain feature fusion information; An image classification module is configured to classify the feature fusion information to obtain a classification result of the user image data; an attribute determination module, configured to determine user attribute information according to the classification result; The operation execution module is configured to determine personalized information according to the user attribute information, and execute the personalized operation corresponding to the personalized information.

9. A vehicle, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the vehicle-mounted system-based data processing method as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method based on the vehicle-mounted system as described in any one of claims 1 to 7 is implemented.

Images