Feature fusion method, device, equipment, medium and program product
By determining the target point cloud features and their target image features from the point cloud feature matrix in intelligent driving for feature mapping and fusion processing, the problem of low fusion efficiency caused by the conversion of image feature matrix to a bird's-eye view perspective in the prior art is solved. Improve the accuracy of object detection in intelligent driving.
Patent Information
- Application Number
- CN202411081198.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-08
AI Technical Summary
The feature fusion method in the prior art requires the conversion of the image feature matrix to a bird's-eye view perspective, resulting in low fusion efficiency.
Determine the target point cloud feature and its corresponding target image features from the point cloud feature matrix, perform feature mapping and fusion processing, obtain the fusion features, and replace the target point cloud feature in the point cloud feature matrix with fusion features to avoid image features The matrix is converted to a bird's-eye view perspective.
It improves the efficiency of feature fusion and enhances the accuracy of target detection, especially the ability to detect traffic elements such as lane lines, signs and ground signs during intelligent driving.
Smart Images

Figure CN118608913B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent driving technology, and in particular to a feature fusion method, device, equipment, medium and program product. Background Art
[0002] In the field of intelligent driving, it is necessary to determine traffic elements such as lane lines, signboards, and ground signs in order to control vehicle driving. In order to improve the accuracy of the determined traffic elements, it is usually necessary to capture images through cameras and collect point cloud data through lidar, and then extract features from road images and point cloud data to obtain image feature matrices and point cloud feature matrices, perform feature fusion, and input the obtained fused feature matrix into the deep learning model for target detection to traffic elements.
[0003] In the prior art, for feature fusion, the image feature matrix is usually converted to a bird's-eye view perspective to obtain a bird's-eye view image feature matrix, and then the bird's-eye view image feature matrix and the point cloud feature matrix are cascaded and input into the convolution layer for processing to obtain a fused feature matrix.
[0004] In summary, existing feature fusion methods require converting the image feature matrix to a bird's-eye view, resulting in low fusion efficiency. Summary of the invention
[0005] The embodiments of the present application provide a feature fusion method, apparatus, device, medium and program product to solve the problem of low fusion efficiency caused by the existing feature fusion method requiring the image features to be converted to a bird's-eye view.
[0006] In a first aspect, an embodiment of the present application provides a feature fusion method, including:
[0007] Determine at least one target point cloud feature from the point cloud feature matrix, perform a point cloud feature processing operation on each of the target point cloud features, and obtain a fusion feature matrix corresponding to the point cloud feature matrix, wherein the point cloud feature processing operation includes:
[0008] Determine the target image features corresponding to the target point cloud features according to the image feature matrix corresponding to the point cloud feature matrix;
[0009] Performing feature mapping and fusion processing on the target point cloud features and the target image features to obtain fusion features corresponding to the target point cloud features;
[0010] The target point cloud features in the point cloud feature matrix are replaced with corresponding fusion features.
[0011] In a possible implementation, determining at least one target point cloud feature from the point cloud feature matrix includes:
[0012] For each point cloud feature that is not 0 in the point cloud feature matrix, calculating the image coordinates of the point cloud feature according to the position of the point cloud feature in the point cloud feature matrix;
[0013] For each point cloud feature that is not 0, if the image coordinates of the point cloud feature belong to a preset image coordinate range, the point cloud feature is used as a target point cloud feature.
[0014] In a possible implementation, determining the target image feature corresponding to the target point cloud feature according to the image feature matrix corresponding to the point cloud feature matrix includes:
[0015] A target image feature corresponding to the target point cloud feature is determined according to the image feature matrix and the image coordinates of the target point cloud feature.
[0016] In a possible implementation, determining the target image feature corresponding to the target point cloud feature according to the image feature matrix and the image coordinates of the target point cloud feature includes:
[0017] If the image coordinates of the target point cloud feature are integers, the image features corresponding to the image coordinates in the image feature matrix are used as the target image features corresponding to the target point cloud feature;
[0018] If the image coordinates of the target point cloud feature are not integers, the adjacent integer coordinates of the image coordinates are selected from the preset image coordinate range; and the target image features corresponding to the target point cloud feature are calculated according to the image features corresponding to each adjacent integer coordinate in the image feature matrix.
[0019] In a possible implementation, the performing feature mapping and fusion processing on the target point cloud feature and the target image feature to obtain a fusion feature corresponding to the target point cloud feature includes:
[0020] Inputting the target point cloud features and the target image features into a preset number of first linear layers for feature mapping, respectively, to obtain mapping point cloud features and mapping image features corresponding to each of the first linear layers;
[0021] For each of the first linear layers, calculating a fusion sub-feature corresponding to the first linear layer according to the mapping point cloud features and the mapping image features corresponding to the first linear layer;
[0022] All fusion sub-features corresponding to the first linear layer are fused to obtain fusion features corresponding to the target point cloud features.
[0023] In a possible implementation, calculating the fusion sub-features corresponding to the first linear layer according to the mapped point cloud features and the mapped image features corresponding to the first linear layer includes:
[0024] Calculate the point cloud weight and image weight corresponding to the first linear layer according to the mapped point cloud features and the mapped image features corresponding to the first linear layer and the second linear layer;
[0025] A weighted sum is performed according to the mapped point cloud features and the mapped image features corresponding to the first linear layer, as well as the point cloud weight and the image weight corresponding to the first linear layer, to obtain a fusion sub-feature corresponding to the first linear layer.
[0026] In a possible implementation, calculating the point cloud weight and the image weight corresponding to the first linear layer according to the mapped point cloud features and the mapped image features corresponding to the first linear layer and the second linear layer includes:
[0027] Cascading the mapping point cloud features and the mapping image features corresponding to the first linear layer to obtain cascade features;
[0028] Inputting the cascade features into the second linear layer for dimensionality reduction processing to obtain a two-dimensional weight vector;
[0029] The weight vector is normalized to obtain a point cloud weight and an image weight corresponding to the first linear layer.
[0030] In a second aspect, an embodiment of the present application provides a feature fusion device, including:
[0031] A processing module, used to determine at least one target point cloud feature from the point cloud feature matrix;
[0032] A fusion module is used to perform a point cloud feature processing operation on each of the target point cloud features to obtain a fusion feature matrix corresponding to the point cloud feature matrix, wherein the point cloud feature processing operation includes:
[0033] Determine the target image features corresponding to the target point cloud features according to the image feature matrix corresponding to the point cloud feature matrix;
[0034] Performing feature mapping and fusion processing on the target point cloud features and the target image features to obtain fusion features corresponding to the target point cloud features;
[0035] The target point cloud features in the point cloud feature matrix are replaced with corresponding fusion features.
[0036] In a third aspect, an embodiment of the present application provides an electronic device, including:
[0037] Processor, memory, communication interface;
[0038] The memory is used to store executable instructions of the processor;
[0039] Wherein, the processor is configured to execute the feature fusion method described in any one of the first aspects by executing the executable instructions.
[0040] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the feature fusion method described in any one of the first aspects.
[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the feature fusion method described in any one of the first aspects.
[0042] The feature fusion method, device, equipment, medium and program product provided in the embodiments of the present application, after determining at least one target point cloud feature from the point cloud feature matrix, determines the target image feature corresponding to the target point cloud feature according to the image feature matrix corresponding to the point cloud feature matrix, and then performs feature mapping and fusion processing on the target point cloud feature and the target image feature to obtain the fusion feature corresponding to the target point cloud feature; and then replaces the target point cloud feature in the point cloud feature matrix with its corresponding fusion feature to obtain a fusion feature matrix. This solution performs feature mapping and fusion processing on the target point cloud feature and its corresponding target image feature to obtain the fusion feature corresponding to the target point cloud feature, and then replaces it in the point cloud feature matrix to obtain the fusion feature matrix, without converting the image feature matrix to a bird's-eye view, thereby improving the fusion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0044] Figure 1a A schematic diagram of the process flow of the first embodiment of the feature fusion method provided in this application;
[0045] Figure 1b A schematic diagram of the fusion feature matrix provided for this application;
[0046] Figure 2 A schematic diagram of the flow chart of the second embodiment of the feature fusion method provided in this application;
[0047] Figure 3a A flowchart of the third embodiment of the feature fusion method provided in this application;
[0048] Figure 3bA schematic diagram of adjacent integer coordinates provided for this application;
[0049] Figure 4a A flowchart of a fourth embodiment of the feature fusion method provided in this application;
[0050] Figure 4b A schematic diagram of the process for determining the characteristics of fusion subunits provided in this application;
[0051] Figure 5 A schematic diagram of the structure of an embodiment of a feature fusion device provided in this application;
[0052] Figure 6 A schematic diagram of the structure of an electronic device provided in this application.
[0053] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0054] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0055] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0056] In order to enable the vehicle to detect traffic elements such as lane lines, signboards, and ground signs in a timely manner during intelligent driving, target detection is required. Before target detection, it is necessary to extract features from road images and point cloud data to obtain image feature matrices and point cloud feature matrices, and then perform feature fusion to obtain fused feature matrices. The fused feature matrix has more semantic and location information. Using the fused feature matrix for target detection can improve the accuracy of target detection.
[0057] In the prior art, for feature fusion, the image feature matrix is usually converted to a bird's-eye view perspective to obtain a bird's-eye view image feature matrix, and then the bird's-eye view image feature matrix is cascaded with the point cloud feature matrix and input into the convolution layer for processing to obtain a fused feature matrix. Since the image feature matrix needs to be converted to a bird's-eye view perspective, the fusion efficiency is low.
[0058] In view of the problems existing in the prior art, the inventors discovered in the process of studying the feature fusion method that in order to improve the fusion efficiency, the target point cloud features and their corresponding target image features can be determined from the image feature matrix and the point cloud feature matrix, and the target point cloud features and their corresponding target image features correspond to the same real position. Then, feature mapping and fusion processing are performed on the target point cloud features and their corresponding target image features to obtain the fusion features corresponding to the target point cloud features, and then the target point cloud features in the point cloud feature matrix are replaced with the corresponding fusion features to obtain the fusion feature matrix. Based on the above inventive concept, the feature fusion scheme in this application is designed.
[0059] The executor of the feature fusion method in the present application may be a computer, or a server, a terminal device, a vehicle, etc. The present application does not limit it and will take a computer as an example for explanation below.
[0060] The following is an example of an application scenario of the feature fusion method provided in this application.
[0061] Exemplarily, in this application scenario, an unmanned vehicle travels on a road, collects point cloud data through a lidar, collects road images through a camera, and then sends the point cloud data and road images to a computer.
[0062] The computer performs feature extraction on the point cloud data to obtain a point cloud feature matrix; and performs feature extraction on the road image to obtain an image feature matrix corresponding to the point cloud feature matrix.
[0063] At least one target point cloud feature is determined from the point cloud feature matrix, and a point cloud feature processing operation is performed on each target point cloud feature to obtain a fusion feature matrix corresponding to the point cloud feature matrix.
[0064] That is, according to the image feature matrix corresponding to the point cloud feature matrix, the target image feature corresponding to the target point cloud feature is determined, and then the target point cloud feature and the target image feature are feature mapped and fused to obtain the fused feature corresponding to the target point cloud feature. Then, the target point cloud feature in the point cloud feature matrix is replaced with its corresponding fused feature to obtain the fused feature matrix.
[0065] The computer performs target detection based on the fused feature matrix and sends the detected traffic elements to the unmanned vehicle. The unmanned vehicle drives according to the detected traffic elements.
[0066] It should be noted that the feature extraction, feature fusion and target detection processes can be performed by an unmanned vehicle.
[0067] It should be noted that the above scenario is only an example of an application scenario provided by an embodiment of the present application. The embodiment of the present application does not limit the actual form of the various devices included in the scenario, nor does it limit the interaction method between the devices. In the specific application of the solution, it can be set according to actual needs.
[0068] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0069] Figure 1a This is a flow chart of the first embodiment of the feature fusion method provided by the present application. The present embodiment of the present application describes the situation in which a computer determines the target point cloud features and their corresponding target image features based on the image feature matrix and the point cloud feature matrix, and performs feature fusion based on the target point cloud features and their corresponding target image features. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware. Figure 1a As shown, the feature fusion method specifically includes the following steps S101 and S102:
[0070] S101: Determine at least one target point cloud feature from a point cloud feature matrix.
[0071] In order to improve the accuracy of target detection, feature fusion is required. The computer extracts features from the acquired point cloud data to obtain a point cloud feature matrix; it extracts features from the acquired road image to obtain an image feature matrix corresponding to the point cloud feature matrix. The point cloud data and the road image are collected at the same location at the same time.
[0072] It should be noted that the image feature matrix is a two-dimensional matrix, each element in the matrix represents an image feature, and each image feature is represented by a vector. The point cloud feature matrix is a three-dimensional matrix, each element in the matrix represents a point cloud feature, and each point cloud feature is represented by a vector.
[0073] It should be noted that when there are multiple road images collected at the same time, feature extraction is performed on each road image to obtain an image feature submatrix of each road image, and then the image feature submatrices are averaged to obtain an image feature matrix.
[0074] In this step, after the computer obtains the image feature matrix and the point cloud feature matrix, it determines at least one target point cloud feature from the point cloud feature matrix.
[0075] The point cloud features whose positions in the point cloud feature matrix can be mapped to the preset image coordinate range and are not 0 are used as target point cloud features. These point cloud features are the features of traffic elements, and the corresponding features can be determined according to the image feature matrix.
[0076] It should be noted that the preset image coordinate range is composed of a preset horizontal coordinate range and a preset vertical coordinate range. The preset horizontal coordinate range and the preset vertical coordinate range can be [0, 128], [0, 256], [0, 1024], etc. The embodiment of the present application does not limit the preset image coordinate range, the preset horizontal coordinate range and the preset vertical coordinate range, and can be determined according to actual conditions.
[0077] It should be noted that since the point cloud feature is represented by a vector, the point cloud feature is determined to be 0 only when each element in the vector is 0, indicating that there is no traffic element at the location corresponding to the point cloud feature. When there is a non-zero element in the vector, the point cloud feature is determined to be non-zero.
[0078] S102: Performing a point cloud feature processing operation on each target point cloud feature to obtain a fusion feature matrix corresponding to the point cloud feature matrix.
[0079] In this step, after the computer obtains the target point cloud features, it performs point cloud feature processing operations on each target point cloud feature to obtain a fusion feature matrix corresponding to the point cloud feature matrix.
[0080] The point cloud feature processing operation for a target point cloud feature includes the following steps S1021, S1022, and S1023:
[0081] S1021, determining the target image feature corresponding to the target point cloud feature according to the image feature matrix corresponding to the point cloud feature matrix.
[0082] In this step, the target image features corresponding to the target point cloud features may be determined, but is not limited to, based on the image coordinates obtained by mapping the image feature matrix and the target point cloud features.
[0083] S1022, performing feature mapping and fusion processing on the target point cloud feature and the target image feature to obtain a fusion feature corresponding to the target point cloud feature.
[0084] In this step, but not limited to, at least one linear layer may be used to perform feature mapping processing on the target point cloud feature and the corresponding target image feature, and then weighted sum is performed to obtain fused sub-features, and then the fused sub-features are fused to obtain fused features.
[0085] S1023, replacing the target point cloud feature in the point cloud feature matrix with its corresponding fusion feature.
[0086] For example, Figure 1b A schematic diagram of the fusion feature matrix provided for this application, such as Figure 1b As shown in the figure, the fusion feature matrix is a three-dimensional matrix, and each three-dimensional square in the figure represents a matrix element, that is, a feature. In the figure, A, B, and C represent the fusion features after replacing the target point cloud features.
[0087] It should be noted that the position of each element in the fusion feature matrix can be represented by three-dimensional coordinates (x, y, z). For elements with the same z coordinate, such elements can be cascaded. This process does not need to be performed during target detection, which can improve the efficiency of subsequent target detection. The z coordinate represents the height.
[0088] The feature fusion method provided in this embodiment determines at least one target point cloud feature from the point cloud feature matrix, determines the target image feature corresponding to the target point cloud feature according to the image feature matrix corresponding to the point cloud feature matrix, and then performs feature mapping and fusion processing on the target point cloud feature and its corresponding target image feature to obtain the fusion feature corresponding to the target point cloud feature; and then replaces the target point cloud feature in the point cloud feature matrix with its corresponding fusion feature to obtain a fusion feature matrix. This solution performs feature mapping and fusion processing on the target point cloud feature and its corresponding target image feature to obtain the fusion feature corresponding to the target point cloud feature, and then replaces it in the point cloud feature matrix to obtain the fusion feature matrix, without converting the image feature matrix to a bird's-eye view, thereby improving the fusion efficiency.
[0089] Figure 2 This is a flow chart of the second embodiment of the feature fusion method provided by the present application. Based on the above embodiment, the present embodiment of the present application describes the situation where a computer determines the target point cloud features. Figure 2 As shown, the feature fusion method specifically includes the following steps:
[0090] S201: For each point cloud feature that is not 0 in the point cloud feature matrix, the image coordinates of the point cloud feature are calculated according to the position of the point cloud feature in the point cloud feature matrix.
[0091] In order to improve the accuracy of subsequent target detection, the target point cloud features can be determined from the point cloud features that are not 0. This is because the point cloud features that are not 0 are the features of traffic elements, and using these features for feature fusion can improve the accuracy of target detection.
[0092] In this step, for each point cloud feature that is not 0 in the point cloud feature matrix, the image coordinates of the point cloud feature are calculated according to the position of the point cloud feature in the point cloud feature matrix.
[0093] Specifically, the formula and formula To calculate the image coordinates of the point cloud features, Indicates the position of the point cloud feature in the point cloud feature matrix, Represents the position of the point cloud feature in the camera coordinate system, is the rotation matrix, is the translation vector, , , , is the camera internal parameter, Image coordinates representing point cloud features.
[0094] S202: For each point cloud feature that is not 0, if the image coordinates of the point cloud feature belong to a preset image coordinate range, the point cloud feature is used as a target point cloud feature.
[0095] In this step, after the computer determines the image coordinates of each non-zero point cloud feature, for each non-zero point cloud feature, if the image coordinates of the point cloud feature belong to the preset image coordinate range, it means that the corresponding feature can be determined through the image feature matrix, then the point cloud feature is used as the target point cloud feature.
[0096] The feature fusion method provided in this embodiment determines the target point cloud features from the point cloud features that are not 0. Since the point cloud features that are not 0 are the features of traffic elements, using these features for feature fusion can improve the accuracy of subsequent target detection.
[0097] Figure 3a This is a flow chart of the third embodiment of the feature fusion method provided by the present application. Based on the above embodiment, the present embodiment of the present application describes the situation of the target image features corresponding to the computer target point cloud features. Figure 3a As shown, the feature fusion method specifically includes the following steps:
[0098] S301: Determine whether the image coordinates of the target point cloud feature are integers; if the image coordinates of the target point cloud feature are integers, execute step S302; if the image coordinates of the target point cloud feature are not integers, execute steps S303-S304.
[0099] In this step, after the computer determines the target point cloud features, it determines the target image features corresponding to the target point cloud features based on the image feature matrix and the image coordinates of the target point cloud features.
[0100] First, it is determined whether the image coordinates of the target point cloud feature are integers, so as to determine the method of obtaining the target image feature matrix.
[0101] It should be noted that, since the image coordinates include a horizontal coordinate and a vertical coordinate, the image coordinates are determined to be integers only when both the horizontal coordinate and the vertical coordinate are integers; when at least one of the horizontal coordinate and the vertical coordinate is not an integer, the image coordinates are determined to be not an integer.
[0102] S302: Using the image feature corresponding to the image coordinates of the target point cloud feature in the image feature matrix as the target image feature corresponding to the target point cloud feature.
[0103] In this step, if the computer determines that the image coordinates of the target point cloud feature are integers, it means that the target image feature can be directly found in the image feature matrix, and the image feature corresponding to the image coordinates of the target point cloud feature in the image feature matrix is used as the target image feature corresponding to the target point cloud feature.
[0104] In one implementation, at least one adjacent integer coordinate of the image coordinates of the target point cloud feature can be selected from a preset image coordinate range, and after determining the image feature corresponding to each selected adjacent integer coordinate in the image feature matrix, the average value of the image feature corresponding to the image coordinate and the image feature corresponding to the adjacent integer coordinate is calculated to obtain the target image feature corresponding to the target point cloud feature, thereby improving feature smoothness.
[0105] It should be noted that adjacent integer coordinates refer to coordinates whose horizontal coordinates and vertical coordinates are both integers, whose horizontal coordinates are less than or equal to 1 away from the horizontal coordinates of the image coordinates, and whose vertical coordinates are less than or equal to 1 away from the vertical coordinates of the image coordinates.
[0106] For example, Figure 3b A schematic diagram of adjacent integer coordinates provided for this application, such as Figure 3bAs shown in the figure, point A represents the image coordinates of a target point cloud feature. The horizontal and vertical coordinates of point A are not integers, so the four points around point A: point B-point E, are the adjacent integer coordinates of point A. Point G represents the image coordinates of a target point cloud feature. The horizontal coordinate of point G is not an integer, but the vertical coordinate is an integer, so the two points on the left and right of point G: point F and point H, are the adjacent integer coordinates of point G.
[0107] S303: Selecting adjacent integer coordinates of the image coordinates of the target point cloud feature from a preset image coordinate range.
[0108] In this step, if the computer determines that the image coordinates of the target point cloud feature are not integers, it means that the target image feature cannot be found directly in the image feature matrix, and it is necessary to select the adjacent integer coordinates of the image coordinates of the target point cloud feature from the preset image coordinate range.
[0109] It should be noted that the number of selected adjacent integer coordinates is at least one.
[0110] S304: Calculate the target image feature corresponding to the target point cloud feature according to the image feature corresponding to each adjacent integer coordinate in the image feature matrix.
[0111] In this step, after obtaining the adjacent integer coordinates of the image coordinates, the computer calculates the target image features corresponding to the target point cloud features according to the image features corresponding to each adjacent integer coordinate in the image feature matrix.
[0112] For each selected adjacent integer coordinate, the distance between the adjacent integer coordinate and the image coordinate is calculated. Then the distance between each adjacent integer coordinate and the image coordinate is normalized to obtain the weight of each adjacent integer coordinate. Then, based on the weight of each adjacent integer coordinate and the image feature corresponding to each adjacent integer coordinate in the image feature matrix, a weighted sum is performed to obtain the target image feature corresponding to the target point cloud feature, which can improve feature smoothness.
[0113] The weighted summation method is: the weight of each adjacent integer coordinate is multiplied by its corresponding image feature and then added.
[0114] In one implementation, all adjacent integer coordinates of the image coordinates of the target point cloud feature are determined from a preset image coordinate range, and then the adjacent integer coordinates closest to the image coordinates are selected, and the image features corresponding to the adjacent integer coordinates in the image feature matrix are used as the target image features corresponding to the target point cloud feature.
[0115] The feature fusion method provided in this embodiment determines the target image features by determining the image coordinates, so that the target point cloud features and the target image features correspond to the same real position, achieving pixel-level alignment, and after fusion, the accuracy of subsequent target detection can be improved.
[0116] Figure 4a This is a flow chart of the fourth embodiment of the feature fusion method provided by the present application. Based on the above embodiment, the present embodiment of the present application describes the situation in which a computer performs feature mapping and fusion processing on the target point cloud features and their corresponding target image features to obtain fusion features corresponding to the target point cloud features. Figure 4a As shown, the feature fusion method specifically includes the following steps:
[0117] S401: Inputting the target point cloud features and the corresponding target image features into a preset number of first linear layers for feature mapping, respectively, to obtain the mapped point cloud features and mapped image features corresponding to each first linear layer.
[0118] In this step, after the computer obtains the target point cloud features and their corresponding target image features, the target point cloud features and the corresponding target image features are respectively input into a preset number of first linear layers for feature mapping to obtain the mapping point cloud features and mapping image features corresponding to each first linear layer.
[0119] By performing feature mapping through different first linear layers, the target point cloud features and their corresponding target image features can be mapped to different feature spaces, so that the mapped point cloud features and mapped image features corresponding to each first linear layer reflect different angles of traffic elements.
[0120] It should be noted that the preset number can be 1, 5, 8, 10, etc. The embodiment of the present application does not limit the preset number and can be determined according to actual conditions.
[0121] S402: For each first linear layer, calculate a fusion sub-feature corresponding to the first linear layer according to the mapping point cloud features and the mapping image features corresponding to the first linear layer.
[0122] In this step, after the computer obtains the mapping point cloud features and the mapping image features corresponding to each first linear layer, for each first linear layer, the computer calculates the fusion sub-features corresponding to the first linear layer according to the mapping point cloud features and the mapping image features corresponding to the first linear layer.
[0123] Specifically, according to the mapped point cloud features and the mapped image features corresponding to the first linear layer, and the second linear layer, the point cloud weight and the image weight corresponding to the first linear layer are calculated.
[0124] That is, the mapping point cloud features and the mapping image features corresponding to the first linear layer are cascaded to obtain cascade features. The cascade features are then input into the second linear layer for dimensionality reduction to obtain a two-dimensional weight vector; the weight vector is then normalized to obtain the point cloud weight and image weight corresponding to the first linear layer.
[0125] Then, according to the mapped point cloud features and the mapped image features corresponding to the first linear layer, as well as the point cloud weights and the image weights corresponding to the first linear layer, weighted summation is performed to obtain the fused sub-features corresponding to the first linear layer.
[0126] The weighted summation method is: calculate the product of the point cloud weight corresponding to the first linear layer and the mapping point cloud feature corresponding to the first linear layer, the product of the image weight corresponding to the first linear layer and the mapping image feature corresponding to the first linear layer, and then add the two products.
[0127] For example, Figure 4b A schematic diagram of the process for determining the characteristics of the fusion subunit provided in this application, such as Figure 4b As shown in the figure, the target point cloud features are passed through the first linear layer to obtain the mapped point cloud features, and the target image features are passed through the first linear layer to obtain the mapped image features. The mapped point cloud features and the mapped image features are cascaded to obtain the cascaded features. The cascaded features are passed through the second linear layer to obtain the weight vector, and the weight vector is normalized to obtain the point cloud weight and the image weight. Then, the mapped point cloud features and the mapped image features are weighted summed with the point cloud weight and the image weight to obtain the fused sub-features.
[0128] S403: Fusing all fusion sub-features corresponding to the first linear layer to obtain fusion features corresponding to the target point cloud features.
[0129] In this step, after the computer obtains the fused sub-features corresponding to each first linear layer, it fuses the fused sub-features corresponding to each first linear layer, that is, cascades them, to obtain the fused features corresponding to the target point cloud features.
[0130] It should be noted that features can be represented by vectors, and the dimension of a feature refers to the number of elements in a vector. In order to make the dimension of the fused feature the same as that of the target point cloud feature, the dimension of the target point cloud feature and its corresponding target image feature is the same, represented by N. The first linear layer will perform dimensionality reduction processing, and the dimension of the mapped point cloud feature and the mapped image feature is N / K, where K is a preset number. Therefore, the dimension of the fused sub-feature is N / K, and the dimension of the fused feature is N. Therefore, it is also necessary to ensure that the preset number can divide the dimension of the target point cloud feature.
[0131] The feature fusion method provided in this embodiment performs feature mapping processing on the target point cloud features and the corresponding target image features through the first linear layer, and then uses the obtained mapped point cloud features and mapped image features to perform feature fusion to obtain fused sub-features, and the fused sub-features are cascaded to obtain fused features. The fused features can better reflect the geometric features and semantic features of traffic elements. The fused features have better generalization capabilities and can improve the accuracy of subsequent target detection.
[0132] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0133] Figure 5 This is a schematic diagram of the structure of an embodiment of the feature fusion device provided in this application. Figure 5 As shown, the feature fusion device 50 includes:
[0134] A processing module 51, used to determine at least one target point cloud feature from the point cloud feature matrix;
[0135] The fusion module 52 is used for:
[0136] Performing a point cloud feature processing operation on each of the target point cloud features to obtain a fusion feature matrix corresponding to the point cloud feature matrix, wherein the point cloud feature processing operation includes:
[0137] Determine the target image features corresponding to the target point cloud features according to the image feature matrix corresponding to the point cloud feature matrix;
[0138] Performing feature mapping and fusion processing on the target point cloud features and the target image features to obtain fusion features corresponding to the target point cloud features;
[0139] The target point cloud features in the point cloud feature matrix are replaced with corresponding fusion features.
[0140] Furthermore, the processing module 51 is specifically used for:
[0141] For each point cloud feature that is not 0 in the point cloud feature matrix, calculating the image coordinates of the point cloud feature according to the position of the point cloud feature in the point cloud feature matrix;
[0142] For each point cloud feature that is not 0, if the image coordinates of the point cloud feature belong to a preset image coordinate range, the point cloud feature is used as a target point cloud feature.
[0143] Furthermore, the fusion module 52 is specifically used for:
[0144] A target image feature corresponding to the target point cloud feature is determined according to the image feature matrix and the image coordinates of the target point cloud feature.
[0145] Furthermore, the fusion module 52 is further configured to:
[0146] If the image coordinates of the target point cloud feature are integers, the image features corresponding to the image coordinates in the image feature matrix are used as the target image features corresponding to the target point cloud feature;
[0147] If the image coordinates of the target point cloud feature are not integers, the adjacent integer coordinates of the image coordinates are selected from the preset image coordinate range; and the target image features corresponding to the target point cloud feature are calculated according to the image features corresponding to each adjacent integer coordinate in the image feature matrix.
[0148] Furthermore, the fusion module 52 is specifically used for:
[0149] Inputting the target point cloud features and the target image features into a preset number of first linear layers for feature mapping, respectively, to obtain mapping point cloud features and mapping image features corresponding to each of the first linear layers;
[0150] For each of the first linear layers, calculating a fusion sub-feature corresponding to the first linear layer according to the mapping point cloud features and the mapping image features corresponding to the first linear layer;
[0151] All fusion sub-features corresponding to the first linear layer are fused to obtain fusion features corresponding to the target point cloud features.
[0152] Furthermore, the fusion module 52 is further configured to:
[0153] Calculate the point cloud weight and image weight corresponding to the first linear layer according to the mapped point cloud features and the mapped image features corresponding to the first linear layer and the second linear layer;
[0154] A weighted sum is performed according to the mapped point cloud features and the mapped image features corresponding to the first linear layer, as well as the point cloud weight and the image weight corresponding to the first linear layer, to obtain a fusion sub-feature corresponding to the first linear layer.
[0155] Furthermore, the fusion module 52 is further configured to:
[0156] Cascading the mapping point cloud features and the mapping image features corresponding to the first linear layer to obtain cascade features;
[0157] Inputting the cascade features into the second linear layer for dimensionality reduction processing to obtain a two-dimensional weight vector;
[0158] The weight vector is normalized to obtain a point cloud weight and an image weight corresponding to the first linear layer.
[0159] The feature fusion device provided in this embodiment is used to execute the technical solution in any of the aforementioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.
[0160] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 6 As shown, the electronic device 60 includes:
[0161] Processor 61, memory 62, and communication interface 63;
[0162] The memory 62 is used to store executable instructions of the processor 61;
[0163] The processor 61 is configured to execute the technical solution in any of the aforementioned method embodiments by executing the executable instructions.
[0164] Optionally, the memory 62 may be independent or integrated with the processor 61 .
[0165] Optionally, when the memory 62 is a device independent of the processor 61, the electronic device 60 may further include:
[0166] The bus 64 , the memory 62 and the communication interface 63 are connected to the processor 61 via the bus 64 and communicate with each other. The communication interface 63 is used to communicate with other devices.
[0167] Optionally, the communication interface 63 may be implemented by a transceiver. The communication interface is used to implement communication between the database access device and other devices (such as a client, a read-write library, and a read-only library). The memory may include a random access memory (RAM) and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.
[0168] The bus 64 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0169] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0170] The electronic device is used to execute the technical solution in any of the aforementioned method embodiments, and its implementation principle and technical effect are similar and will not be repeated here.
[0171] An embodiment of the present application also provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the technical solution provided by any of the aforementioned method embodiments is implemented.
[0172] An embodiment of the present application also provides a computer program product, including a computer program, which is used to implement the technical solution provided by any of the aforementioned method embodiments when executed by a processor.
[0173] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A feature fusion method, It is characterized in that include: Determine at least one target point cloud feature from the point cloud features that are not zero and are included in the point cloud feature matrix, perform a point cloud feature processing operation on each of the target point cloud features, and obtain a fusion feature matrix corresponding to the point cloud feature matrix, wherein the point cloud feature processing operation includes: Determine the target image feature corresponding to the target point cloud feature according to the image feature matrix corresponding to the point cloud feature matrix and the image coordinates of the target point cloud feature; Inputting the target point cloud features and the target image features into a preset number of first linear layers for feature mapping, respectively, to obtain mapping point cloud features and mapping image features corresponding to each of the first linear layers; For each of the first linear layers, calculating a fusion sub-feature corresponding to the first linear layer according to the mapping point cloud features and the mapping image features corresponding to the first linear layer; Fusing all fusion sub-features corresponding to the first linear layer to obtain a fusion feature corresponding to the target point cloud feature; Replacing the target point cloud features in the point cloud feature matrix with corresponding fusion features; Wherein, determining the target image feature corresponding to the target point cloud feature according to the image feature matrix corresponding to the point cloud feature matrix and the image coordinates of the target point cloud feature comprises: If the image coordinates of the target point cloud feature are integers, the image features corresponding to the image coordinates in the image feature matrix are used as the target image features corresponding to the target point cloud feature; If the image coordinates of the target point cloud feature are not integers, the adjacent integer coordinates of the image coordinates are selected from the preset image coordinate range; and the target image features corresponding to the target point cloud feature are calculated according to the image features corresponding to each adjacent integer coordinate in the image feature matrix.
2. The method according to claim 1, It is characterized in that The step of determining at least one target point cloud feature from the point cloud features that are not zero and are included in the point cloud feature matrix includes: For each point cloud feature that is not 0 in the point cloud feature matrix, calculating the image coordinates of the point cloud feature according to the position of the point cloud feature in the point cloud feature matrix; For each point cloud feature that is not 0, if the image coordinates of the point cloud feature belong to a preset image coordinate range, the point cloud feature is used as the target point cloud feature.
3. The method according to claim 1, It is characterized in that The calculating, according to the mapped point cloud features and the mapped image features corresponding to the first linear layer, the fusion sub-features corresponding to the first linear layer includes: Calculate the point cloud weight and image weight corresponding to the first linear layer according to the mapped point cloud features and the mapped image features corresponding to the first linear layer and the second linear layer; A weighted sum is performed according to the mapped point cloud features and the mapped image features corresponding to the first linear layer, as well as the point cloud weight and the image weight corresponding to the first linear layer, to obtain a fusion sub-feature corresponding to the first linear layer.
4. The method according to claim 3, It is characterized in that The step of calculating the point cloud weight and the image weight corresponding to the first linear layer according to the mapped point cloud features and the mapped image features corresponding to the first linear layer and the second linear layer includes: Cascading the mapping point cloud features and the mapping image features corresponding to the first linear layer to obtain cascade features; Inputting the cascade features into the second linear layer for dimensionality reduction processing to obtain a two-dimensional weight vector; The weight vector is normalized to obtain a point cloud weight and an image weight corresponding to the first linear layer.
5. A feature fusion device, It is characterized in that include: A processing module, used for determining at least one target point cloud feature from the point cloud features that are not zero and are included in the point cloud feature matrix; A fusion module is used to perform a point cloud feature processing operation on each of the target point cloud features to obtain a fusion feature matrix corresponding to the point cloud feature matrix, wherein the point cloud feature processing operation includes: Determine the target image feature corresponding to the target point cloud feature according to the image feature matrix corresponding to the point cloud feature matrix and the image coordinates of the target point cloud feature; Inputting the target point cloud features and the target image features into a preset number of first linear layers for feature mapping, respectively, to obtain mapping point cloud features and mapping image features corresponding to each of the first linear layers; For each of the first linear layers, calculating a fusion sub-feature corresponding to the first linear layer according to the mapping point cloud features and the mapping image features corresponding to the first linear layer; Fusing all fusion sub-features corresponding to the first linear layer to obtain a fusion feature corresponding to the target point cloud feature; Replacing the target point cloud features in the point cloud feature matrix with corresponding fusion features; Wherein, determining the target image feature corresponding to the target point cloud feature according to the image feature matrix corresponding to the point cloud feature matrix and the image coordinates of the target point cloud feature comprises: If the image coordinates of the target point cloud feature are integers, the image features corresponding to the image coordinates in the image feature matrix are used as the target image features corresponding to the target point cloud feature; If the image coordinates of the target point cloud feature are not integers, the adjacent integer coordinates of the image coordinates are selected from the preset image coordinate range; and the target image features corresponding to the target point cloud feature are calculated according to the image features corresponding to each adjacent integer coordinate in the image feature matrix.
6. An electronic device, It is characterized in that include: Processor, memory, communication interface; The memory is used to store executable instructions of the processor; Wherein, the processor is configured to execute the feature fusion method described in any one of claims 1 to 4 by executing the executable instructions.
7. A readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the feature fusion method according to any one of claims 1 to 4 is implemented.
8. A computer program product, It is characterized in that It comprises a computer program, which is used to implement the feature fusion method described in any one of claims 1 to 4 when executed by a processor.
Citation Information
Patent Citations
Real-time fusion method and device based on point cloud and image data
CN112348941A
Backlight water surface target identification method, system and device fusing laser point cloud and RGB image data
CN114648686A
Image imaging method and device
CN115150564A
Rapid unmanned aerial vehicle landing area detection method based on multi-sensor fusion
CN116524177A
Semantic component attitude estimation method based on deep learning
CN117218343A
