Image processing method and device, vehicle and computer readable storage medium

Abstract

The invention provides an image processing method and device, a vehicle and a computer readable storage medium, and the method is applied to the technical field of computers, and comprises the steps: carrying out the partitioning processing of a to-be-processed image, obtaining a plurality of sub-images, extracting the first image features of the plurality of sub-images, and obtaining the first image features of the plurality of sub-images; respectively mapping the plurality of first image features to a basic coordinate system so as to reconstruct the plurality of first image features and obtain second image features of the plurality of sub-images in the basic coordinate system; and according to the plurality of sub-images, the respective second image features of the plurality of sub-images are corrected to obtain respective third image features of the plurality of sub-images, and the target task is executed based on the plurality of third image features. The key information in the image can be accurately captured, the general features of the image are effectively expressed, the general features of the image are applied to tasks such as image classification, target detection and semantic segmentation, the generalization ability of the image features is enhanced, and the performance and efficiency of image processing tasks are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to an image processing method, device, vehicle and computer-readable storage medium in the field of computer technology. Background Art

[0002] At present, image feature extraction methods are widely used in many fields. However, current image feature extraction methods cannot fully capture the key information in the image, resulting in weak expression ability of image features, which in turn affects the application of image features in subsequent tasks (such as image classification tasks, object detection tasks, semantic segmentation tasks, etc.). For example, in the object detection task, the object detection model may not be able to accurately frame the position of the target object based on the image features, resulting in detection frame offset or missed detection.

[0003] Therefore, it is urgent to propose effective solutions to the above problems. Summary of the invention

[0004] The present application provides an image processing method, device, vehicle and computer-readable storage medium. The present application divides the image to be processed into blocks, extracts the features of each sub-image, and maps the features of all sub-images to a low-dimensional space coordinate system for feature reconstruction to achieve noise reduction, eliminate redundant information, etc., which can accurately capture the key information in the image and effectively express the features of the image, so as to facilitate the application of image features to subsequent tasks (such as image classification tasks, target detection tasks, semantic segmentation tasks, etc.), thereby significantly improving the performance and efficiency of image processing tasks.

[0005] In a first aspect, an image processing method is provided, which includes: performing block processing on an image to be processed to obtain multiple sub-images; performing feature extraction on the multiple sub-images to obtain first image features of each of the multiple sub-images; for each sub-image, mapping the first image feature of the sub-image to a basic coordinate system to reconstruct the first image feature of the sub-image to obtain a second image feature of the sub-image located in the basic coordinate system; wherein the dimension of the first image feature of the sub-image is higher than the dimension of the second image feature of the sub-image; correcting the second image feature of the sub-image according to the sub-image to obtain a third image feature of the sub-image; and executing a target task based on the first modal feature; wherein the first modal feature includes the third image feature of each of the multiple sub-images.

[0006] Through the above technical scheme, the image to be processed is processed in blocks to obtain multiple sub-images, the first image features of each of the multiple sub-images are extracted, and the first image features of the multiple sub-images are respectively mapped to the basic coordinate system to reconstruct the first image features of each of the multiple sub-images, and the second image features of each of the multiple sub-images located in the basic coordinate system are obtained; the second image features of each of the multiple sub-images are corrected according to the multiple sub-images to obtain the third image features of each of the multiple sub-images, and the target task is performed based on the third image features of each of the multiple sub-images. The present application divides the image to be processed into blocks, extracts the features of each block of sub-images, and maps the features of all sub-images to a low-dimensional space coordinate system for feature reconstruction to achieve noise reduction, eliminate redundant information, etc., so as to accurately capture the key information in the image and effectively express the features of the image. Since the first image features of multiple sub-images are mapped to the basic coordinate system for dimensionality reduction in the present application, the new features (second image features) generated after removing the redundant information are mutually orthogonal, that is, there is no linear correlation between the second image features, which reduces the dependency between the features. Therefore, the second image features of the image extracted in the present application are universal. When performing different tasks (such as image classification tasks, target detection tasks, semantic segmentation tasks, etc.) based on the same image to be processed, the solution provided in this application can be used to extract the common features of the image, and the common features of the image can be applied to subsequent different tasks. The application range of image features is wider, and the generalization ability of image features is enhanced. That is, there is no need to extract different image features according to different tasks, which significantly improves the performance and efficiency of image processing tasks.

[0007] In combination with the first aspect, in some possible implementations, feature extraction is performed on multiple sub-images to obtain the first image features of each of the multiple sub-images, including: flattening each of the multiple sub-images into a one-dimensional vector to obtain a data matrix corresponding to the multiple sub-images; calculating the mean of the data matrix to obtain the mean vector corresponding to the multiple sub-images; obtaining the data matrix after averaging based on the difference between the data matrix and the mean vector; calculating the covariance matrix of the data matrix after averaging; calculating the eigenvalues ​​of the covariance matrix and the eigenvectors corresponding to the eigenvalues ​​of the covariance matrix; constructing an eigenvector matrix based on the eigenvalues ​​and the eigenvectors; constructing a principal component space based on the eigenvector matrix; mapping the data matrix after averaging to the principal component space to obtain the first image features of each of the multiple sub-images. Based on the above technical solution, by preprocessing multiple sub-images to obtain the data matrix after averaging corresponding to the multiple sub-images, and mapping the data matrix after averaging from the original high-dimensional space to the new low-dimensional space, it is possible to eliminate redundant information in the sub-images, reduce the data dimension, and extract the image features of the multiple sub-images.

[0008] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, before mapping the first image feature of the sub-image to the basic coordinate system, the image processing method also includes: obtaining M sample images; wherein M≥2; for each sample image, performing block processing on the sample image to obtain N sample sub-images of the sample image; wherein N≥2; performing feature extraction on the N sample sub-images corresponding to each of the M sample images to obtain the sixth image feature corresponding to each of the M×N sample sub-images; clustering the sixth image features corresponding to each of the M×N sample sub-images to obtain feature vectors of multiple basic features; wherein the feature vector of the basic feature is the vector of the cluster center; and constructing a basic coordinate system based on the feature vectors of the multiple basic features.

[0009] Based on the above technical solution, by acquiring multiple sample images, dividing the multiple sample images into blocks, multiple sample sub-images, and extracting image features of the multiple sample sub-images, clustering analysis is performed on the image features of the multiple sample sub-images, and characteristic vectors of multiple basic features corresponding to the multiple sample sub-images are obtained. These characteristic vectors serve as coordinate vectors of cluster centers, representing the main patterns and structures of the data. The characteristic vectors of multiple basic features serve as coordinate axes to construct a low-dimensional space coordinate system (basic coordinate system). Since the dimension of the basic coordinate system is determined according to the number of basic features, the basic coordinate system can not only retain important information of the image, but also simplify the representation of image information, that is, it can effectively represent and reconstruct image information.

[0010] In combination with the first aspect and the above implementation, in some possible implementations, executing the target task based on the first modal feature includes: obtaining a feature relationship between the first modal feature and the second modal feature; wherein the second modal feature includes data features other than image features, and the data features other than image features include audio features and / or text features, and the feature relationship is a linear relationship or a nonlinear relationship; based on the feature relationship, the first modal feature and the second modal feature are fused to obtain a fused feature; and the target task is executed based on the fused feature. Based on the above technical solution, by fusing the first modal feature and the second modal feature, a multimodal fusion feature can be obtained, so as to execute the target task based on the fused feature.

[0011] In combination with the first aspect and the above-mentioned implementation, in some possible implementations, when the feature relationship between the first modal feature and the second modal feature is a linear relationship, the first modal feature and the second modal feature are fused based on the feature relationship to obtain the fused feature, including: obtaining a weight value corresponding to the first modal feature and a weight value corresponding to the second modal feature; performing a weighted operation on the first modal feature and the second modal feature based on the weight value corresponding to the first modal feature and the weight value corresponding to the second modal feature to obtain the fused feature. Based on the above-mentioned technical solution, when the feature relationship between the first modal feature and the second modal feature is a linear relationship, the fused feature can be quickly obtained by performing a linear weighted combination of the first modal feature and the second modal feature, thereby improving data processing efficiency.

[0012] In combination with the first aspect and the above-mentioned implementation, in some possible implementations, when the feature relationship between the first modal feature and the second modal feature is a nonlinear relationship, the first modal feature and the second modal feature are fused based on the feature relationship to obtain the fused feature, including: obtaining the weight value corresponding to the first modal feature and the weight value corresponding to the second modal feature; performing weighted operations on the first modal feature and the second modal feature based on the weight value corresponding to the first modal feature and the weight value corresponding to the second modal feature to obtain a weighted feature vector; performing a nonlinear transformation on the weighted feature vector according to an activation function to obtain a fused feature. Based on the above-mentioned technical solution, when the feature relationship between the first modal feature and the second modal feature is a nonlinear relationship, an activation function is introduced to perform a nonlinear transformation on the weighted feature vector, so that the fused feature has nonlinear characteristics, thereby increasing the expression ability of the fusion model.

[0013] In combination with the first aspect and the above implementations, in some possible implementations, executing the target task based on the fused features includes: identifying the task category of the target task; and controlling the vehicle according to the task category and the fused features.

[0014] In a second aspect, an image processing device is provided, the image processing device comprising:

[0015] A processing module is used to perform block processing on the image to be processed to obtain multiple sub-images; perform feature extraction on the multiple sub-images to obtain first image features of each of the multiple sub-images; for each sub-image, map the first image feature of the sub-image to a basic coordinate system to reconstruct the first image feature of the sub-image to obtain a second image feature of the sub-image located in the basic coordinate system; wherein the dimension of the first image feature of the sub-image is higher than the dimension of the second image feature of the sub-image; and correct the second image feature of the sub-image according to the sub-image to obtain a third image feature of the sub-image;

[0016] A control module is used to perform a target task based on a first modal feature; wherein the first modal feature includes a third image feature of each of the plurality of sub-images.

[0017] In a third aspect, a vehicle is provided, the vehicle comprising: a memory for storing executable program code;

[0018] A processor is used to call and run executable program code from a memory so that the vehicle executes the image processing method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0019] In a fourth aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the image processing method in the first aspect or any possible implementation of the first aspect.

[0020] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the image processing method in the above-mentioned first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic flow chart of an image processing method provided in an embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown;

[0023] Figure 3 A schematic structural diagram of a vehicle provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0025] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0026] At present, image feature extraction methods are widely used in many fields. In existing multimodal data processing tasks, image feature processing methods include: convolutional neural networks, vision transformers (ViT), and self-supervised learning. Since these models are usually trained on tasks in specific fields, the shallow features of the extracted image features are usually more general and suitable for most image processing tasks, while the deep features are task-specific. In order to achieve the best performance on specific tasks, it is usually necessary to adjust and fine-tune the model architecture appropriately. That is, the image features extracted by the current image feature extraction methods are not very general, which affects the application of image features in subsequent tasks (such as image classification tasks, object detection tasks, semantic segmentation tasks, etc.).

[0027] Based on the problems existing in the above-mentioned related technologies, the embodiments of the present application provide an image processing method, device, vehicle and computer-readable storage medium. The present application divides the image to be processed into blocks, extracts the features of each sub-image, and maps the features of all sub-images to a low-dimensional space coordinate system for feature reconstruction to achieve noise reduction, eliminate redundant information and other processing, and can accurately capture the key information in the image and effectively express the features of the image. Since the first image features of multiple sub-images are mapped to the basic coordinate system for dimensionality reduction in the present application, the new features (second image features) generated after removing the redundant information are mutually orthogonal, that is, there is no linear correlation between the second image features, which reduces the dependency between the features. Therefore, the second image features of the image extracted in the present application are universal. When performing different tasks (such as image classification tasks, target detection tasks, semantic segmentation tasks, etc.) according to the same image to be processed, the universal features of the image can be extracted through the scheme provided by the present application, and the universal features of the image can be applied to subsequent different tasks. The application range of the image features is wider, and the generalization ability of the image features is enhanced, that is, there is no need to extract different image features according to different tasks, which significantly improves the performance and efficiency of the image processing tasks.

[0028] The following is an embodiment of an image processing method provided in an embodiment of the present application.

[0029] Figure 1 A schematic flow chart of an image processing method provided by an embodiment of the present application is shown, such as Figure 1 As shown, the embodiment of the present application provides an image processing method, which includes the following scheme:

[0030] S110: Divide the image to be processed into blocks to obtain multiple sub-images.

[0031] S120: Extract features from the multiple sub-images to obtain first image features of the multiple sub-images.

[0032] S130: For each sub-image, map the first image feature of the sub-image to the basic coordinate system to reconstruct the first image feature of the sub-image and obtain the second image feature of the sub-image in the basic coordinate system.

[0033] S140: Correct the second image feature of the sub-image according to the sub-image to obtain a third image feature of the sub-image.

[0034] S150: Execute a target task based on the first modal feature.

[0035] It should be understood that when extracting image features of an image, the image to be processed is divided into blocks, which can capture the local details of the image to be processed and ensure that the information of each area in the image to be processed can be captured. When extracting image features, the image to be processed is divided into several small blocks (usually called "windows" or "pixel blocks") to improve the accuracy and efficiency of feature extraction. Since each small block only focuses on the pixel values ​​around it, it can more accurately capture the local texture, edge and other features of the image, thereby avoiding the blur caused by large-scale calculations; and smaller blocks help to retain more details, such as the edges, textures and color changes of objects, which is very useful for distinguishing similar objects or detecting subtle differences; through block division, large images are decomposed into many independent subtasks, reducing the overall computing requirements, and for complex models and large data sets, the execution speed of the algorithm can be improved. When extracting image features of an image, the image to be processed is processed in blocks, which utilizes the spatial redundancy and local consistency of the image, making feature extraction more targeted and flexible, and enhancing the performance of image recognition and analysis.

[0036] Exemplarily, the image to be processed is divided into small blocks of fixed size, each of which is a sub-image. For example, each sub-image is a block of 16×16 pixels. Dividing the image to be processed into blocks can ensure that the information of each area (i.e., local area) in the image to be processed is processed independently.

[0037] After the image to be processed is divided into blocks to obtain multiple sub-images, features of the multiple sub-images can be extracted through principal component analysis, convolutional neural network, self-supervised learning, etc. to obtain first image features of each of the multiple sub-images.

[0038] Since principal component analysis, convolutional neural network and self-supervised learning cannot fully capture the key information in the image, that is, the first image features of each of the multiple sub-images currently obtained cannot accurately reflect the features of the multiple sub-images. Therefore, the present application maps the first image feature of the sub-image to a preset basic coordinate system to reduce the dimension and remove redundancy of the first image feature of the sub-image, that is, calculate the representation coefficient of the first image feature of the sub-image in the basic coordinate system to obtain the second image feature of the sub-image in the basic coordinate system. Among them, the preset basic coordinate system is a low-dimensional space coordinate system, and the space in which it is located belongs to a low-dimensional space. The dimension of the space in which the sub-image is located is higher than the space in which the basic coordinate system is located, that is, the dimension of the second image feature of the sub-image obtained by mapping the first image feature of the sub-image to the preset basic coordinate system is lower than the dimension of the first image feature of the sub-image. Calculating the representation coefficient of the first image feature in the basic coordinate system can be understood as finding a set of coefficients through optimization methods such as least squares method, gradient descent, conjugate gradient method, etc., so that the first image feature can be represented by a linear combination of basis vectors in the basic coordinate system.

[0039] In order to ensure that the features of the sub-images are captured completely and accurately, the present application further minimizes the loss function through optimization algorithms such as gradient descent and adaptive moment estimation to reduce the error between the second image features of each sub-image and each sub-image, thereby obtaining the third image features of each sub-image.

[0040] The third image feature of each sub-image is obtained, and the third image features of multiple sub-images are used as the first modal features, that is, all local features of the image to be processed are obtained, and each local feature can accurately represent the corresponding local area, that is, the first modal feature can accurately represent the global image to be processed. Then, the first modal feature of the same image to be processed can be used to process various target tasks, such as image classification tasks, target detection tasks, semantic segmentation tasks, etc.

[0041] Through the above technical solution, the image to be processed is divided into blocks, the features of each sub-image are extracted, and the features of all sub-images are mapped to a low-dimensional space coordinate system for feature reconstruction to achieve noise reduction, eliminate redundant information and other processing, which can accurately capture the key information in the image and effectively express the features of the image. Since the first image features of multiple sub-images are mapped to the basic coordinate system for dimensionality reduction in the present application, the new features (second image features) generated after removing the redundant information are mutually orthogonal, that is, there is no linear correlation between the second image features, which reduces the dependency between the features. Therefore, the second image features of the image extracted in the present application are universal. When performing different tasks (such as image classification tasks, target detection tasks, semantic segmentation tasks, etc.) according to the same image to be processed, the universal features of the image can be extracted through the solution provided by the present application, and the universal features of the image can be applied to subsequent different tasks. The application range of image features is wider, and the generalization ability of image features is enhanced, that is, there is no need to extract different image features according to different tasks, which significantly improves the performance and efficiency of image processing tasks.

[0042] As can be seen from the above, after the image to be processed is divided into blocks and multiple sub-images are obtained, the present application needs to perform feature extraction on the multiple sub-images to obtain the first image features of each of the multiple sub-images. The specific process of feature extraction on multiple sub-images is exemplarily described below.

[0043] In a possible implementation, the extracting features of the multiple sub-images to obtain the first image features of the multiple sub-images includes:

[0044] Flatten each of the multiple sub-images into a one-dimensional vector to obtain a data matrix corresponding to the multiple sub-images;

[0045] Calculate the mean of the data matrix to obtain the mean vectors corresponding to multiple sub-images;

[0046] According to the difference between the data matrix and the mean vector, the data matrix after removing the mean is obtained;

[0047] Calculate the covariance matrix of the data matrix after removing the mean;

[0048] Calculate the eigenvalues ​​of the covariance matrix and the eigenvectors of the covariance matrix corresponding to the eigenvalues;

[0049] Construct an eigenvector matrix based on eigenvalues ​​and eigenvectors;

[0050] Construct the principal component space based on the eigenvector matrix;

[0051] The data matrix after de-averaging is mapped to the principal component space to obtain the first image features of each of the multiple sub-images.

[0052] Principal component analysis is a dimensionality reduction algorithm that extracts the main information of data by reducing noise and eliminating redundant information. The essence of principal component analysis is equivalent to mapping data from the original high-dimensional space to a new low-dimensional space. Each dimension in the new low-dimensional space is a principal component, representing a major change direction of the data.

[0053] Exemplarily, in an embodiment of the present application, a principal component analysis method is used to extract features from multiple sub-images to obtain first image features of each of the multiple sub-images. Specifically, first, the multiple sub-images are preprocessed, and each of the multiple sub-images is converted into a vector to obtain multiple vectors corresponding to the multiple sub-images. In the process of flattening the image into a one-dimensional vector, it is usually necessary to normalize the image data, that is, to subtract the mean from each pixel value so that the data mean is zero, so as to eliminate the offset of the data and improve the effect of feature extraction. For example, assuming image A, image B and image C, each image is flattened to obtain a vector of length 4, image A is flattened to obtain vector a[1,2,3,4], image B is flattened to obtain vector b[5,6,7,8], and image C is flattened to obtain vector c[9,10,11,12]. Arrange the multiple vectors in columns to form a data matrix corresponding to the multiple sub-images. The data matrix corresponding to the multiple sub-images refers to the same matrix composed of the vectors corresponding to the multiple sub-images, that is, the same matrix composed of all the vectors corresponding to all the sub-images. For example, vector a, vector b, and vector c obtain the data matrices corresponding to image A, image B, and image C Furthermore, the mean of each row of the data matrix is ​​calculated to obtain the mean vector corresponding to the multiple sub-images. For example, the mean of each row of the data matrix X is calculated to obtain the mean vector Further, the mean vector is subtracted from the data matrix to obtain the data matrix after the mean is removed. Among them, the mean removal of the data matrix can eliminate the translation effect of the data, simplify the calculation, and improve the numerical stability. For example, the mean vector Y is subtracted from the data matrix X to obtain the data matrix after the mean is removed

[0054] Before mapping the data matrix after the mean value is removed corresponding to multiple sub-images from the original high-dimensional space to the new low-dimensional space, a new low-dimensional space needs to be constructed. Specifically, first, the covariance matrix of the data matrix after the mean value is removed is calculated. By calculating the covariance matrix, the internal structure of the data can be deeply understood. On the one hand, the covariance matrix can measure the linear relationship between different features. Through the covariance matrix, we can understand which features are positively correlated, negatively correlated or uncorrelated; on the other hand, the data structure can be identified through the covariance matrix. The eigenvalues ​​of the covariance matrix and the eigenvectors corresponding to the eigenvalues ​​of the covariance matrix can be obtained through the characteristic equation. Among them, the eigenvalue represents the data variance in the direction of the corresponding eigenvector. The larger eigenvalue indicates that the data in this direction has a larger change, that is, the direction contains more information, that is, the eigenvector corresponding to this direction is more important. Therefore, when constructing the low-dimensional space coordinate system, the eigenvalues ​​are sorted in order from large to small, and the eigenvectors are arranged accordingly. Usually, the eigenvectors corresponding to the first k largest eigenvalues ​​are selected as the principal components to form the eigenvector matrix. Selecting the first k largest eigenvalues ​​can retain most of the variance, thereby achieving effective dimensionality reduction. The eigenvectors corresponding to the first k largest eigenvalues ​​are orthogonal (i.e., perpendicular to each other). The eigenvectors corresponding to the first k largest eigenvalues ​​serve as the coordinate axes of the coordinate system to form a new low-dimensional space coordinate system. Each eigenvector corresponds to a principal component, indicating the main changes in the image features in that direction.

[0055] After completing the construction of the new low-dimensional space, the averaged data matrices corresponding to the multiple sub-images are projected onto these eigenvectors, that is, the averaged data matrices are mapped from the original high-dimensional space to the new low-dimensional space to obtain the first image features of each of the multiple sub-images.

[0056] Based on the above technical solution, by preprocessing multiple sub-images, the data matrix corresponding to the multiple sub-images after averaging is obtained, and the data matrix after averaging is mapped from the original high-dimensional space to the new low-dimensional space, it is possible to eliminate redundant information in the sub-images, reduce the data dimension, and extract image features of multiple sub-images.

[0057] Exemplarily, in the embodiment of the present application, a convolutional neural network can also be used to extract features from multiple sub-images to obtain first image features of each of the multiple sub-images. Specifically, the convolution layer of the feature extraction model is used to extract local features of the sub-image, and the local features are nonlinearly transformed through an activation function to obtain a fourth image feature; the pooling layer of the feature extraction model is used to downsample the fourth image feature to obtain a fifth image feature; wherein the dimension of the fifth image feature is smaller than the dimension of the fourth image feature; the fully connected layer of the feature extraction model is used to convert the fourth image feature and the fifth image feature into one-dimensional vectors of the same length, and the one-dimensional vectors are nonlinearly transformed through an activation function to obtain the first image feature.

[0058] Exemplarily, a convolutional neural network is mainly composed of a convolution layer, a pooling layer, an activation function, a fully connected layer, etc. Among them, the convolution layer extracts local features by applying multiple convolution kernels to slide on the input data. Each convolution kernel can capture different types of local features, such as edges, textures, etc. The output of the convolution layer is a set of feature maps. The feature map represents the local features of the input data. Exemplarily, the convolution operation formula can be expressed as:

[0059] X l+1 =f(W l *X l +b l )

[0060] Among them, X l represents the input of the convolutional layer, W l is the convolution kernel weight of the convolution layer, * represents the convolution operation, b l is the bias term, f is the activation function, X 1+1 is the output feature map of the convolutional layer.

[0061] Since the input of each convolutional layer is a linear function of the output of the upper convolutional layer, the entire convolutional neural network is a multi-layer linear transformation. The superposition of multiple layers of linear transformations is still a linear transformation. The ability of linear transformation is very limited and can only represent linear relationships, while many problems and data in application scenarios are nonlinear. Pure linear transformation cannot approximate complex nonlinear functions. Therefore, in an embodiment of the present application, a nonlinear function can be introduced as an activation function. Common nonlinear activation functions include ReLU (Rectified Linear Unit), Sigmoid, Tanh, etc. The role of the nonlinear activation function is to perform a nonlinear transformation on the input signal, so that the entire convolutional neural network has a stronger expressive ability. In addition, by introducing appropriate nonlinear activation functions (such as ReLU) in model training, the gradient vanishing problem can be effectively alleviated. Exemplarily, the activation function ReLU formula can be expressed as:

[0062] f(x)=max(0,x)

[0063] Exemplarily, in an embodiment of the present application, a convolutional layer is used to extract local features of each of the multiple sub-images, and to perform nonlinear transformation on the multiple groups of local features through an activation function to obtain fourth image features of each of the multiple sub-images.

[0064] Since the fourth image feature extracted by the convolution layer retains the local information of the input data, but has a high dimension and high computational complexity, it needs to be input to the pooling layer for further processing.

[0065] The role of the pooling layer is to reduce the dimension of the feature map output by the convolutional layer by downsampling, thereby reducing the computational complexity and retaining the most important data features. Commonly used pooling operations include Max Pooling and Average Pooling.

[0066] Exemplarily, the maximum pooling formula can be expressed as:

[0067] X pool =max(X l+1 )

[0068] The maximum pooling layer is used to extract the maximum value in the feature map and retain the most significant features in the local area.

[0069] Exemplarily, in the embodiment of the present application, the pooling layer is used to downsample the fourth image feature of each of the multiple sub-images, reduce the dimension of the fourth image feature, and obtain the fifth image feature of each of the multiple sub-images, wherein the dimension of the fifth image feature is smaller than the dimension of the fourth image feature.

[0070] The function of the fully connected layer is to convert the feature maps extracted by the convolution layer and the pooling layer into the final output of the convolutional neural network. Exemplarily, in an embodiment of the present application, the fully connected layer is used to convert the fourth image features of each of the multiple sub-images and the fifth image features of each of the multiple sub-images into multiple groups of one-dimensional vectors of the same length, and perform nonlinear transformation of the multiple groups of one-dimensional vectors of the same length through an activation function to obtain the first image features of each of the multiple sub-images. Based on the above technical solution, the convolution layer extracts local features through a sliding window operation, while the pooling layer downsamples these features to retain the most important information, which can effectively capture the spatial structure and hierarchical relationship in the image.

[0071] It should be understood that after extracting the first image feature of the sub-image, the present application needs to map the first image feature of the sub-image to the basic coordinate system, but before mapping the first image feature of the sub-image to the basic coordinate system, the present application needs to construct the basic coordinate system. The specific process of constructing the basic coordinate system is exemplarily described below.

[0072] In a possible implementation, before mapping the first image feature of the sub-image to the basic coordinate system, the image processing method further includes:

[0073] Get M sample images;

[0074] For each sample image, the sample image is divided into blocks to obtain N sample sub-images of the sample image; features are extracted from the N sample sub-images corresponding to each of the M sample images to obtain sixth image features corresponding to each of the M×N sample sub-images;

[0075] Clustering the sixth image features corresponding to each of the M×N sample sub-images to obtain feature vectors of multiple basic features;

[0076] A basic coordinate system is constructed according to feature vectors of multiple basic features.

[0077] Exemplarily, before mapping the first image feature of the sub-image to the basic coordinate system, it is necessary to construct a basic coordinate system. Specifically, first, multiple sample images are obtained, for example, M sample images are obtained; wherein M≥2. In order to ensure the breadth and representativeness of the data set, sample images can be collected from multiple different sources so that the sample images cover various scenes, lighting conditions, shooting angles and object changes, thereby improving the generalization ability and robustness of the feature extraction model. The sample images are standardized, such as resizing, color correction and noise removal, to ensure the consistency of the input data. Further, each sample image is divided into small blocks of a fixed size to obtain multiple sample sub-images to capture the local details of the sample image. For example, each sample image in the M sample images is divided into N sample sub-images to obtain M×N sample sub-images; wherein N≥2. Then, feature extraction is performed on the M×N sample sub-images by means of principal component analysis, convolutional neural network and self-supervised learning, the dimension is reduced, redundant information is removed, and the key features of the sample sub-images are retained, so as to obtain the sixth image features corresponding to each of the M×N sample sub-images. Finally, cluster analysis is performed on the extracted M×N sixth image features to obtain feature vectors of multiple basic features, wherein the feature vectors of multiple basic features are the coordinate vectors of the cluster centers of multiple clusters obtained by clustering the M×N sixth image features. The feature vectors of multiple basic features serve as coordinate axes to form a low-dimensional space coordinate system to obtain a basic coordinate system; wherein the dimension of the low-dimensional space where the basic coordinate system is located is determined according to the number of feature vectors of the basic features. The basic coordinate system can effectively represent and reconstruct image information, providing a general representation framework. Based on the above technical solution, by acquiring multiple sample images, dividing the multiple sample images into blocks, multiple sample sub-images, and extracting image features of the multiple sample sub-images, clustering analysis is performed on the image features of the multiple sample sub-images, and characteristic vectors of multiple basic features corresponding to the multiple sample sub-images are obtained. These characteristic vectors serve as coordinate vectors of cluster centers, representing the main patterns and structures of the data. The characteristic vectors of multiple basic features serve as coordinate axes to construct a low-dimensional space coordinate system (basic coordinate system). Since the dimension of the basic coordinate system is determined according to the number of basic features, the basic coordinate system can not only retain important information of the image, but also simplify the representation of image information, that is, it can effectively represent and reconstruct image information.

[0078] In a possible implementation, the performing of the target task based on the first modal feature includes:

[0079] Obtaining a feature relationship between the first modal feature and the second modal feature;

[0080] The first modality feature and the second modality feature are fused based on the feature relationship to obtain a fused feature;

[0081] Perform the target task based on the fused features.

[0082] Among them, the second modal features include data features other than image features. Specifically, the data features other than image features include, for example, audio features and / or text features. The feature relationship is a linear relationship or a nonlinear relationship. By fusing image features with audio features and / or text features, a comprehensive feature after multimodal fusion can be obtained to perform image classification tasks, target detection tasks, semantic segmentation tasks, etc.

[0083] Exemplarily, the task of identifying the driver's emotions can be performed by fusing image features and audio features. For example, the driver's facial image and the driver's voice are obtained, and image features and audio features are extracted respectively, wherein image features can capture changes in facial expressions; audio features can capture information such as the tone and speech speed of the voice. By fusing the image features and audio features, the fused comprehensive features can be obtained, and the driver's emotions can be accurately identified based on the fused comprehensive features. The task of generating an accident report can be performed by fusing image features and text features. For example, photos of the accident scene and descriptions of witnesses are obtained, and image features and text features are extracted respectively. By fusing the image features and text features, the fused comprehensive features can be obtained, and the accident report can be quickly generated based on the fused comprehensive features.

[0084] Based on the above technical solution, by fusing the first modal feature and the second modal feature through the feature relationship between the first modal feature and the second modal feature, a multimodal fusion feature can be obtained, so as to perform the target task based on the fusion feature.

[0085] In a possible implementation, when the feature relationship between the first modal feature and the second modal feature is a linear relationship, the fusion feature obtained by fusing the first modal feature and the second modal feature based on the feature relationship includes:

[0086] Obtaining a weight value corresponding to the first modal feature and a weight value corresponding to the second modal feature;

[0087] A weighted operation is performed on the first modal feature and the second modal feature based on the weight value corresponding to the first modal feature and the weight value corresponding to the second modal feature to obtain a fusion feature.

[0088] Exemplarily, the first modality feature and the second modality feature may be fused through a fusion model of logistic regression or random forest.

[0089] For example, the fusion formula can be expressed as formula (1):

[0090] y ′ =w1y1+w2y2 (1)

[0091] Among them, y′ Represents fusion features. y1 is the first modality feature, y2 is the second modality feature, w1 is the weight of the first modality feature, and w2 is the weight of the second modality feature.

[0092] Formula (1) directly performs a linear weighted combination of features. The fusion model has a simple structure and high processing efficiency. Therefore, formula (1) is suitable for simple tasks, real-time systems and resource-constrained scenarios.

[0093] In a possible implementation, when the feature relationship between the first modal feature and the second modal feature is a nonlinear relationship, the fusion feature obtained by fusing the first modal feature and the second modal feature based on the feature relationship includes:

[0094] Obtaining a weight value corresponding to the first modal feature and a weight value corresponding to the second modal feature;

[0095] Performing a weighted operation on the first modal feature and the second modal feature based on the weight value corresponding to the first modal feature and the weight value corresponding to the second modal feature to obtain a weighted feature vector;

[0096] The weighted feature vector is nonlinearly transformed according to the activation function to obtain the fused feature.

[0097] For example, the fusion formula can be expressed as formula (2):

[0098] y ′ =g(w1y1+w2y2) (2)

[0099] Among them, y l Represents fusion features. y2 is the first modality feature, y2 is the second modality feature, w1 is the first modality feature weight, w2 is the second modality feature weight, and g is the activation function.

[0100] Since the activation function g is introduced in formula (2), the fused features can have nonlinear characteristics, which increases the expressive power of the fusion model. Therefore, formula (2) is suitable for complex tasks that require strong nonlinear fitting capabilities and feature interactions.

[0101] The specific process of executing the target task based on the fusion features is exemplified below.

[0102] In a possible implementation, the above-mentioned execution of the target task based on the fusion feature includes:

[0103] Identify the task category of the target task;

[0104] Vehicle control is performed based on the task category and fused features.

[0105] Exemplarily, when the target image is a facial image of the driver, the task category of the target task is identified. If the task category of the target task is an automatic reminder task, such as a fatigue driving reminder task, it is determined based on the fusion features whether the driver is in a fatigue driving state; if so, a prompt message is output, and the prompt message is used to remind the driver to rest.

[0106] Fatigue driving may cause the driver to be slow to react and make mistakes in judgment, thus causing traffic accidents. Therefore, it is very important to monitor the driver's fatigue state in real time and remind him in time. In the embodiment of the present application, the fusion features of image features and audio features can be used to determine whether the driver is in a fatigue driving state, and output prompt information when the driver is detected to be in a fatigue driving state. Exemplarily, the driver's facial image is continuously captured by a vehicle-mounted camera, and the acquired facial image is subjected to feature extraction. The image features can capture changes in facial expressions, such as the driver's blinking frequency, the degree of opening and closing of the eyes, and the head posture, etc. At the same time, the driver's voice is continuously acquired through the vehicle-mounted microphone, such as the driver's breathing frequency and yawning frequency. The image features and audio features are fused into a comprehensive feature vector, and the fused feature vector is input into a pre-trained classification model to determine whether the driver is in a fatigue driving state, wherein the classification model includes, for example, a support vector machine, a random forest, a deep learning model, etc. Exemplarily, when the driver's blinking frequency is greater than a first threshold value, and the driver's yawning frequency is greater than a second threshold value, the driver is considered to be in a fatigue driving state. When the driver is judged to be in a fatigue driving state, the vehicle outputs a prompt message to remind the driver to rest as soon as possible to ensure driving safety. For example, the prompt message can be output in a variety of ways, for example, a prompt tone "Please note that you may have been driving fatigued, please rest in time" can be played through the vehicle audio, or a prompt message such as "Fatigue driving, please rest in time" can be displayed on the vehicle display screen, or the driver can be prompted by the vibration of the steering wheel or seat.

[0107] Exemplarily, when the target image is an image of the external environment of the vehicle, the task category of the target task is identified. If the task category of the target task is an automatic obstacle avoidance task, such as an automatic obstacle avoidance task, obstacles are detected based on fused features. According to the obstacle detection result, the vehicle is controlled to perform at least one of deceleration, steering and braking to avoid the obstacle.

[0108] Automatic obstacle avoidance is an important part of autonomous driving technology, which aims to ensure the safe driving of the vehicle by detecting obstacles around the vehicle in real time and taking corresponding obstacle avoidance measures according to the detection results. In the embodiment of the present application, the fusion of image features and text features can be used to detect obstacles, and the vehicle can be controlled to perform corresponding obstacle avoidance operations according to the detection results. Exemplarily, the image of the external environment is continuously captured by the on-board camera, and the feature extraction of the acquired image of the external environment is performed. The image features can capture the shape, size, position, etc. of the obstacle. At the same time, the data detected by sensors such as radar and lidar configured by the vehicle is obtained, and the sensor data is converted into text features. The image features can capture the speed and direction of the obstacle, and the image features and text features are fused into a comprehensive feature vector. The fused feature vector is input into the pre-trained detection model, and the category, position and speed of the obstacle are output. According to the obstacle detection result, the vehicle is controlled to perform at least one operation of deceleration, steering and braking to avoid the obstacle. For example: if the obstacle is detected to be a slow-moving vehicle, the vehicle can be controlled to slow down; if the obstacle is detected to be a stone, the vehicle can be controlled to turn; if the obstacle is detected to be a pedestrian crossing the road, the vehicle can be controlled to perform emergency braking to ensure that the vehicle safely avoids the obstacle.

[0109] Through the above technical solution, the image to be processed is divided into blocks, the features of each sub-image are extracted, and the features of all sub-images are mapped to a low-dimensional space coordinate system for feature reconstruction to achieve noise reduction, eliminate redundant information, etc., which can accurately capture the key information in the image and effectively express the features of the image. Since the first image features of multiple sub-images are mapped to the basic coordinate system for dimensionality reduction in the present application, the new features (second image features) generated after removing the redundant information are mutually orthogonal, that is, there is no linear correlation between the second image features, which reduces the dependency between the features. Therefore, the second image features of the image extracted in the present application are universal. When performing different tasks (such as image classification tasks, target detection tasks, semantic segmentation tasks, etc.) according to the same image to be processed, the universal features of the image can be extracted through the present application, and the universal features of the image can be applied to subsequent different tasks. The application range of image features is wider, and the generalization ability of image features is enhanced, that is, there is no need to extract different image features according to different tasks, which significantly improves the performance and efficiency of image processing tasks.

[0110] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the image processing device of the present application, please refer to the embodiment of the driving behavior recognition method of the present application.

[0111] Figure 2The schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown. The image processing device 200 includes:

[0112] 210 processing module, used for performing block processing on the image to be processed to obtain multiple sub-images; performing feature extraction on the multiple sub-images to obtain first image features of each of the multiple sub-images; for each sub-image, mapping the first image feature of the sub-image to a basic coordinate system to reconstruct the first image feature of the sub-image to obtain a second image feature of the sub-image located in the basic coordinate system; wherein the dimension of the first image feature of the sub-image is higher than the dimension of the second image feature of the sub-image; and correcting the second image feature of the sub-image according to the sub-image to obtain a third image feature of the sub-image;

[0113] 220 A control module is used to execute a target task based on a first modal feature; wherein the first modal feature includes a third image feature of each of the plurality of sub-images.

[0114] In a possible implementation, the processing module 210 is specifically configured to:

[0115] Flatten each of the multiple sub-images into a one-dimensional vector to obtain a data matrix corresponding to the multiple sub-images;

[0116] Calculate the mean of the data matrix to obtain the mean vectors corresponding to multiple sub-images;

[0117] According to the difference between the data matrix and the mean vector, the data matrix after removing the mean is obtained;

[0118] Calculate the covariance matrix of the data matrix after removing the mean;

[0119] Calculate the eigenvalues ​​of the covariance matrix and the eigenvectors of the covariance matrix corresponding to the eigenvalues;

[0120] Construct an eigenvector matrix based on eigenvalues ​​and eigenvectors;

[0121] Construct the principal component space based on the eigenvector matrix;

[0122] The data matrix after de-averaging is mapped to the principal component space to obtain the first image features of each of the multiple sub-images.

[0123] In a possible implementation, the processing module 210 is further configured to:

[0124] Obtain M sample images; where M ≥ 2;

[0125] For each sample image, the sample image is divided into blocks to obtain N sample sub-images of the sample image; wherein N ≥ 2;

[0126] Extract features from the N sample sub-images corresponding to the M sample images to obtain sixth image features corresponding to the M×N sample sub-images;

[0127] Clustering the sixth image features corresponding to each of the M×N sample sub-images to obtain feature vectors of multiple basic features; wherein the feature vector of the basic feature is the vector of the cluster center;

[0128] A basic coordinate system is constructed according to feature vectors of multiple basic features.

[0129] In a possible implementation, the control module 220 is specifically configured to:

[0130] Acquire a feature relationship between the first modal feature and the second modal feature; wherein the second modal feature includes data features other than image features, the data features other than image features include audio features and / or text features, and the feature relationship is a linear relationship or a nonlinear relationship;

[0131] The first modality feature and the second modality feature are fused based on the feature relationship to obtain a fused feature;

[0132] Perform the target task based on the fused features.

[0133] In a possible implementation, the control module 220 is specifically configured to:

[0134] Obtaining a weight value corresponding to the first modal feature and a weight value corresponding to the second modal feature;

[0135] A weighted operation is performed on the first modal feature and the second modal feature based on the weight value corresponding to the first modal feature and the weight value corresponding to the second modal feature to obtain a fusion feature.

[0136] In a possible implementation, the control module 220 is specifically configured to:

[0137] Obtaining a weight value corresponding to the first modal feature and a weight value corresponding to the second modal feature;

[0138] Performing a weighted operation on the first modal feature and the second modal feature based on the weight value corresponding to the first modal feature and the weight value corresponding to the second modal feature to obtain a weighted feature vector;

[0139] The weighted feature vector is nonlinearly transformed according to the activation function to obtain the fused feature.

[0140] In a possible implementation, the control module 220 is specifically configured to:

[0141] Identify the task category of the target task;

[0142] Vehicle control is performed based on the task category and fused features.

[0143] Figure 3 It is a structural schematic diagram of a vehicle provided in an embodiment of the present application.

[0144] For example, Figure 3 As shown, the vehicle 300 includes: a memory 301 and a processor 302, wherein the memory 301 stores an executable program code 3011, and the processor 302 is used to call and execute the executable program code 3011 to perform an image processing method.

[0145] In this embodiment, the functional modules of the vehicle can be divided according to the above method example. For example, each functional module can be corresponded, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0146] In the case of dividing each functional module according to each function, the vehicle may include: an acquisition module, a processing module, a control module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0147] The vehicle provided in this embodiment is used to execute the above-mentioned image processing method, so the same effect as the above-mentioned implementation method can be achieved.

[0148] In the case of an integrated unit, the vehicle may include a processing module and a storage module. The processing module may be used to control and manage the actions of the vehicle. The storage module may be used to support the vehicle in executing relevant program codes and data.

[0149] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0150] This embodiment further provides a computer-readable storage medium, in which a computer program code is stored. When the computer program code is executed on a computer, the computer executes the above-mentioned related method steps to implement an image processing method in the above-mentioned embodiment.

[0151] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement an image processing method in the above-mentioned embodiment.

[0152] In addition, the vehicle provided in the embodiments of the present application may specifically be a chip, component or module, and the vehicle may include a connected processor and memory; wherein the memory is used to store instructions, and when the vehicle is running, the processor may call and execute instructions so that the chip executes an image processing method in the above-mentioned embodiments.

[0153] The vehicle, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding image processing method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding image processing method provided above, and will not be repeated here.

[0154] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0155] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0156] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The image processing method comprises: The image to be processed is processed into blocks to obtain multiple sub-images; Extracting features from the plurality of sub-images to obtain first image features of the plurality of sub-images; For each sub-image, mapping the first image feature of the sub-image to a basic coordinate system to reconstruct the first image feature of the sub-image, and obtaining the second image feature of the sub-image located in the basic coordinate system; wherein the dimension of the first image feature of the sub-image is higher than the dimension of the second image feature of the sub-image; Correcting the second image feature of the sub-image according to the sub-image to obtain a third image feature of the sub-image; The target task is performed based on the first modality feature; wherein the first modality feature includes the third image feature of each of the plurality of sub-images.

2. The image processing method according to claim 1, characterized in that: The extracting features of the plurality of sub-images to obtain first image features of each of the plurality of sub-images comprises: Flatten each of the plurality of sub-images into a one-dimensional vector to obtain a data matrix corresponding to the plurality of sub-images; Calculating the mean of the data matrix to obtain mean vectors corresponding to the plurality of sub-images; Obtaining a data matrix after removing the mean value according to the difference between the data matrix and the mean vector; Calculating the covariance matrix of the data matrix after de-meaning; Calculating eigenvalues ​​of the covariance matrix and eigenvectors of the covariance matrix corresponding to the eigenvalues; Constructing an eigenvector matrix according to the eigenvalues ​​and the eigenvectors; Constructing a principal component space based on the eigenvector matrix; The data matrix after the mean value is removed is mapped to the principal component space to reconstruct the data matrix after the mean value is removed to obtain the first image features of each of the plurality of sub-images.

3. The image processing method according to claim 1, characterized in that: Also includes: Obtain M sample images; where M ≥ 2; For each sample image, the sample image is processed into blocks to obtain N sample sub-images of the sample image; wherein N≥2; Performing feature extraction on the N sample sub-images corresponding to each of the M sample images to obtain sixth image features corresponding to each of the M×N sample sub-images; Clustering the sixth image features corresponding to each of the M×N sample sub-images to obtain feature vectors of a plurality of basic features; wherein the feature vector of the basic feature is a vector of a cluster center; The basic coordinate system is constructed according to the feature vectors of the multiple basic features.

4. The image processing method according to claim 1, characterized in that: The performing the target task based on the first modal feature includes: Acquire a feature relationship between the first modal feature and the second modal feature; wherein the second modal feature includes data features other than image features, the data features other than image features include audio features and / or text features, and the feature relationship is a linear relationship or a nonlinear relationship; Based on the feature relationship, the first modal feature and the second modal feature are fused to obtain a fused feature; The target task is performed based on the fused features.

5. The image processing method according to claim 4, characterized in that: In the case where the feature relationship is a linear relationship, the fusing the first modal feature and the second modal feature based on the feature relationship to obtain the fused feature includes: Obtaining a weight value corresponding to the first modal feature and a weight value corresponding to the second modal feature; The first modal feature and the second modal feature are weightedly calculated based on the weight value corresponding to the first modal feature and the weight value corresponding to the second modal feature to obtain the fusion feature.

6. The image processing method according to claim 4, characterized in that: In the case where the feature relationship is a nonlinear relationship, the fusing the first modal feature and the second modal feature based on the feature relationship to obtain the fused feature includes: Obtaining a weight value corresponding to the first modal feature and a weight value corresponding to the second modal feature; Performing a weighted operation on the first modal feature and the second modal feature based on the weight value corresponding to the first modal feature and the weight value corresponding to the second modal feature to obtain a weighted feature vector; The weighted feature vector is nonlinearly transformed according to an activation function to obtain the fusion feature.

7. The image processing method according to claim 4, characterized in that: The performing of the target task based on the fusion feature comprises: Identifying a task category of the target task; The vehicle is controlled according to the task category and the fused features.

8. An image processing device, characterized in that: The image processing device comprises: A processing module, used for performing block processing on the image to be processed to obtain a plurality of sub-images; performing feature extraction on the plurality of sub-images to obtain first image features of each of the plurality of sub-images; for each sub-image, mapping the first image feature of the sub-image to a basic coordinate system to reconstruct the first image feature of the sub-image to obtain a second image feature of the sub-image located in the basic coordinate system; correcting the second image feature of the sub-image according to the sub-image to obtain a third image feature of the sub-image; wherein the dimension of the first image feature of the sub-image is higher than the dimension of the second image feature of the sub-image; A control module is used to perform a target task based on a first modal feature; wherein the first modal feature includes a third image feature of each of the plurality of sub-images.

9. A vehicle, characterized in that: The vehicle comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the vehicle executes the image processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the image processing method according to any one of claims 1 to 7 is implemented.

Images