Fusion methods, apparatuses, and storage media

Abstract

This application discloses a fusion method, apparatus, and storage medium. The fusion method includes: acquiring first image features at multiple scales corresponding to image data, and acquiring first point cloud features at multiple scales corresponding to radar point cloud data; fusing the first image features and first point cloud features at each scale to obtain first fusion features corresponding to each scale; wherein the first fusion feature at the first scale is obtained by fusing the first image features and first point cloud features at the first scale using reference information, the reference information including at least one of the first point cloud features at the second scale and the first fusion feature at the second scale, the first scale and the second scale being two adjacent scales among multiple scales, and the first scale being lower than the second scale; using each first fusion feature, a fusion result for a target detection task is obtained. This method can improve the fusion effect of image data and radar point cloud data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a fusion method, apparatus and storage medium. Background Technology

[0002] In the fields of intelligent transportation and urban security, with the ever-increasing demand for all-weather, high-precision environmental perception, 3D target detection technology has become a core support for ensuring public safety and traffic management. Typical application scenarios include security monitoring systems in key areas such as urban road checkpoints, highway entrances and exits, bridges and tunnels, park perimeters, transportation hubs, and urban squares. In these application scenarios, security monitoring systems need to continuously identify targets such as vehicles, pedestrians, and non-motorized vehicles to achieve accurate positioning, speed estimation, and behavior analysis, in order to support intelligent security tasks such as traffic violation detection, vehicle wrong-way driving alarms, pedestrian intrusion warnings, non-motorized vehicle violation identification, queue length statistics, and congestion monitoring.

[0003] Traditional security monitoring systems rely heavily on image data, but they are prone to failure under complex conditions such as nighttime, fog, or strong backlighting. Radar, with its ability to actively emit electromagnetic waves, can penetrate harsh environments and provide stable distance and velocity observations, effectively compensating for the shortcomings of visual modalities. However, due to the differences in spatial sampling density, feature representation, and geometric structure between image data and radar point cloud data, improving the cross-modal fusion effect of image data and radar point cloud data has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a fusion method, apparatus, and computer-readable storage medium that can improve the fusion effect of image data and radar point cloud data.

[0005] To address the aforementioned technical problems, this application provides a fusion method comprising: acquiring first image features at multiple scales corresponding to image data, and acquiring first point cloud features at multiple scales corresponding to radar point cloud data; fusing the first image features and first point cloud features at each scale to obtain first fusion features corresponding to each scale; wherein the first fusion feature at the first scale is obtained by fusing the first image features and first point cloud features at the first scale using reference information, the reference information including at least one of the first point cloud features at the second scale and the first fusion feature at the second scale, the first scale and the second scale being two adjacent scales among multiple scales, and the first scale being lower than the second scale; and using each first fusion feature to obtain a fusion result for a target detection task.

[0006] In one embodiment, the reference information includes first point cloud features at a second scale. The reference information is used to fuse the first image features and the first point cloud features at the first scale to obtain a first fused feature at the first scale. This includes: acquiring the first image features at the first scale as a query feature at the first scale; and acquiring the first point cloud features at the first scale and the first point cloud features at the second scale together as key features and value features at the first scale; using the query features, key features, and value features at the first scale to determine a second fused feature at the first scale; and using the second fused feature at the first scale to determine the first fused feature at the first scale.

[0007] In one embodiment, the reference information further includes a first fusion feature at a second scale. Before determining the second fusion feature at the first scale using the query feature, key feature, and value feature at the first scale, the method further includes: determining a first attention weight and a positional offset at the first scale using the first image feature, the first point cloud feature at the first scale, and the first fusion feature at the second scale. Determining the second fusion feature at the first scale using the query feature, key feature, and value feature at the first scale includes: determining the second fusion feature at the first scale using the query feature, key feature, value feature, first attention weight, and positional offset at the first scale.

[0008] In one embodiment, determining a first fusion feature at a first scale using a second fusion feature at a first scale includes: determining the sum of the second fusion feature at the first scale and a first image feature at the first scale as the first fusion feature at the first scale.

[0009] In one embodiment, before fusing the first image features and the first point cloud features at the first scale using reference information to obtain the first fused features at the first scale, the method further includes: adjusting the resolution corresponding to the reference information so that the resolution corresponding to the reference information is the same as the resolution corresponding to the first scale.

[0010] In one embodiment, a fusion result for a target detection task is obtained by utilizing each first fusion feature, including: fusing each first fusion feature to obtain a target fusion feature; and using the target fusion feature as the fusion result.

[0011] In one embodiment, fusing the first fusion features to obtain the target fusion feature includes: performing feature concatenation on the first fusion features to obtain concatenated features; and performing convolution processing on the concatenated features to obtain the target fusion feature.

[0012] In one embodiment, before fusing the first fusion features to obtain the target fusion feature, the method further includes: adjusting the resolution of each first fusion feature so that the resolution of each first fusion feature is the same.

[0013] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, including a memory and a processor coupled to each other, wherein the memory stores program instructions; and the processor is used to execute the program instructions stored in the memory to implement the above-mentioned fusion method.

[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing program instructions that can be executed by a processor to implement the above-mentioned fusion method.

[0015] The above scheme acquires first image features at multiple scales corresponding to image data and first point cloud features at multiple scales corresponding to radar point cloud data; it then fuses the first image features and first point cloud features at each scale to obtain first fusion features corresponding to each scale; finally, it uses these first fusion features to obtain the fusion result for the target detection task. Specifically, the first fusion feature at the first scale is obtained by fusing the first image features and first point cloud features at the first scale using reference information. The reference information includes at least one of the first point cloud features at the second scale and the first fusion feature at the second scale. The first and second scales are two adjacent scales among multiple scales, with the first scale being lower than the second scale. In this method, the reference information is passed from high scale to low scale. The first point cloud features and first fusion features at high scales are passed as reference information to the low scales, guiding the low-scale fusion process. This allows the low-scale fusion process to reference the semantic information provided by the high scale, improving the accuracy of the first fusion features at low scales, thereby improving the fusion effect of image data and radar point cloud data, and ultimately improving the accuracy of the target detection result. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an embodiment of the fusion method provided in this application; Figure 2 This is a flowchart illustrating another embodiment of the fusion method provided in this application; Figure 3 This is a schematic diagram of the framework of an embodiment of the fusion device provided in this application; Figure 4 This is a schematic diagram of the framework of an embodiment of the electronic device provided in this application; Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] It should be noted that the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. The term "multiple" in this application means at least two, such as two, three, etc. The term "several" in this application means at least two. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0019] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the fusion method provided in this application. It should be noted that if substantially the same result is achieved, the method of this application does not necessarily require further elaboration. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, the method includes the following steps: S11: Obtain first image features at multiple scales corresponding to the image data, and obtain first point cloud features at multiple scales corresponding to the radar point cloud data.

[0020] In step S11, image data is acquired via a camera, and radar point cloud data is acquired via radar. For example, the radar is a millimeter-wave radar.

[0021] In step S11, each scale corresponds to one first image feature and one first point cloud feature. Furthermore, the resolution of the first image feature and the first point cloud feature at the same scale is the same. Multiple scales can be two, three, or four scales, etc., and this embodiment does not specifically limit the number of multiple scales.

[0022] Understandably, the larger the scale, the lower the corresponding resolution, the larger the receptive field, the fewer the geometric details, and the stronger the corresponding semantics (e.g., it can accurately identify the target category, such as a car or a pedestrian); the smaller the scale, the higher the corresponding resolution, the smaller the receptive field, the more geometric details, and the weaker the corresponding semantics (it is difficult to distinguish the target category and is easily affected by noise).

[0023] S12: Fuse the first image features and the first point cloud features at each scale to obtain the first fused features corresponding to each scale; wherein, the first fused feature at the first scale is obtained by fusing the first image features and the first point cloud features at the first scale using reference information, the reference information including at least one of the first point cloud features at the second scale and the first fused feature at the second scale, the first scale and the second scale are two adjacent scales among multiple scales and the first scale is lower than the second scale.

[0024] In step S12, the reference information is transferred in order from high scale to low scale. For example, if there are multiple scales including scale a, scale b and scale c, where scale a is higher than scale b and scale b is higher than scale c, then the reference information of scale a is transferred to scale b, and the reference information of scale b is transferred to scale c.

[0025] S13: Utilize the first fusion features at each scale to obtain the fusion results for the target detection task.

[0026] In some implementations, the first fusion features of all scales in multiple scales are fused to obtain a fusion result; or, the first fusion features of each first scale in multiple scales are fused to obtain a fusion result; or, the first fusion feature of one of the first scales in multiple scales is used as the fusion result.

[0027] In one embodiment, after obtaining the fusion result, the fusion result is further used to perform a target detection task. For example, the target detection task can be three-dimensional target detection. After inputting the obtained fusion result into the DETR3D detection head, the detection result corresponding to the three-dimensional target detection is obtained.

[0028] In this embodiment, first image features at multiple scales corresponding to image data and first point cloud features at multiple scales corresponding to radar point cloud data are acquired. The first image features and first point cloud features at each scale are fused to obtain first fused features corresponding to each scale. Using these first fused features, a fusion result for the target detection task is obtained. Specifically, the first fused feature at the first scale is obtained by fusing the first image features and first point cloud features at the first scale using reference information. The reference information includes at least one of the first point cloud features at the second scale and the first fused feature at the second scale. The first scale and the second scale are two adjacent scales among multiple scales, with the first scale being lower than the second scale. In this method, the reference information is passed from high scale to low scale. The first point cloud features and first fused features at high scales are passed as reference information to low scales, guiding the fusion process at low scales. This allows the fusion process at low scales to reference the semantic information provided by high scales, improving the accuracy of the first fused features at low scales, thereby improving the fusion effect of image data and radar point cloud data, and ultimately improving the accuracy of the target detection result.

[0029] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the fusion method provided in this application. It should be noted that if substantially the same result is achieved, the method of this application does not necessarily require further elaboration. Figure 2 The illustrated process sequence is limited. For example... Figure 2 As shown, the method includes the following steps: S21: The image data is processed using an image encoder to obtain second image features at multiple scales corresponding to the image data; and the radar point cloud data is processed using a point cloud encoder to obtain second point cloud features at multiple scales corresponding to the radar point cloud data.

[0030] In one embodiment, the image encoder includes a first feature extraction network and a first feature pyramid network. In this embodiment, the image encoder encodes image data to obtain second image features at multiple scales corresponding to the image data, further including the following steps: Step 1: Use the first feature extraction network to extract features from the image data to obtain the third image features at multiple levels corresponding to the image data.

[0031] The first feature extraction network can extract features at different levels. By using the first feature extraction network to extract features from image data, several levels of image features corresponding to the image data can be obtained. From these several levels of image features, multiple levels of image features are selected as the third image features corresponding to the image data at multiple levels. For example, the several levels of image features include... and Choose from and The third image feature of these three levels.

[0032] For example, the first feature extraction network is a ResNet network.

[0033] Step 2: Use the first feature pyramid network to process the third image features at multiple levels corresponding to the image data to obtain the second image features at multiple scales corresponding to the image data.

[0034] For example, and The third image features from these three levels are input into the first feature pyramid network to obtain... and Second image features at these three scales.

[0035] In one embodiment, the point cloud encoder includes a second feature extraction network and a second feature pyramid network. In this embodiment, the point cloud encoder processes radar point cloud data to obtain second point cloud features at multiple scales corresponding to the radar point cloud data, further including the following steps: Step 1: Use the second feature extraction network to extract features from the radar point cloud data to obtain the third point cloud features at multiple levels corresponding to the radar point cloud data.

[0036] The second feature extraction network can extract features at different levels. By using the second feature extraction network to extract features from radar point cloud data, several levels of point cloud features corresponding to the radar point cloud data can be obtained. Multiple levels of point cloud features are selected from these multiple levels of features as the third level of point cloud features corresponding to the radar point cloud data. For example, the multiple levels of point cloud features include... and Choose from and The third point cloud feature of these three levels.

[0037] For example, the second feature extraction network is VoxelNet.

[0038] Step 2: Use the second feature pyramid network to process the third point cloud features of multiple levels corresponding to the radar point cloud data to obtain the second point cloud features of multiple scales corresponding to the radar point cloud data.

[0039] For example, and The third point cloud features of these three levels are input into the second feature pyramid network to obtain... and The second point cloud features at these three scales.

[0040] In step S21, both the image encoder and the point cloud encoder use a Feature Pyramid Network (FPN). The Feature Pyramid Network can construct a multi-scale feature pyramid from the features provided by the backbone network (for the image branch, the backbone network is the first feature extraction network; for the radar branch, the backbone network is the second feature extraction network) to generate multi-scale features with different resolutions but enhanced semantic information.

[0041] S22: Map the second image features at multiple scales corresponding to the image data and the second point cloud features at multiple scales corresponding to the radar point cloud data to the target coordinate system to obtain the first image features at multiple scales corresponding to the image data and the first point cloud features at multiple scales corresponding to the radar point cloud data.

[0042] Since image features are two-dimensional and point cloud features are three-dimensional, and their coordinate systems are different, they cannot be directly fused. Only after the second image features corresponding to multiple scales of image data and the second point cloud features corresponding to multiple scales of radar point cloud data are uniformly mapped to the same target coordinate system can feature fusion and interaction be performed.

[0043] In one embodiment, the target coordinate system is the BEV (Bird's-Eye View) coordinate system. LSS (Lift-Splat-Shoot) is used to uniformly map the second image features at multiple scales corresponding to the image data and the second point cloud features at multiple scales corresponding to the radar point cloud data to the BEV coordinate system. LSS is based on the sensor's geometric projection model and maps image feature points... and radar point cloud A mapping relationship is established in three-dimensional space, and the features of the two modalities are unified into the BEV coordinate system through differentiable grid sampling and frustum partitioning, generating a BEV feature pyramid with high semantic hierarchy. The content of LSS can be found in known related technologies, and will not be explained in detail here.

[0044] For example, and Second image features at these three scales and and After unifying the second point cloud features at these three scales to the BEV coordinate system, we obtain... and The first image features at these three scales, and, are obtained and The first point cloud features at these three scales. Among them, and These are the first image features and the first point cloud features at the B3 scale, respectively. and These are the first image features and the first point cloud features at the B4 scale, respectively. and These are the first image features and the first point cloud features at scale B5, respectively. Scale B5 is higher than scale B4, and scale B4 is higher than scale B3.

[0045] For example, the resolution corresponding to the B3 scale is The resolution corresponding to the B4 scale is The resolution corresponding to the B5 scale is The B3 scale has high resolution, can preserve details of small objects, and is suitable for precise localization at close range, but has weaker semantics; the B4 scale can balance semantic and geometric details, and is suitable for mid-range target perception; the B5 scale has a large receptive field and strong semantics, but low resolution, and is suitable for modeling distant targets and scene semantics.

[0046] S23: Fuse the first image features and the first point cloud features at each scale to obtain the first fused features corresponding to each scale.

[0047] In step S23, for the highest scale among multiple scales, no reference information is used, and only the first image feature and the first point cloud feature of the highest scale are fused to obtain the first fused feature of the highest scale.

[0048] In one embodiment, DCA (Deformable Cross Attention) is used to fuse the highest-scale first image feature and the first point cloud feature to obtain the highest-scale first fused feature. This embodiment includes the following steps: Step 1: Obtain the highest-scale first image feature as the highest-scale query feature, and obtain the highest-scale first point cloud feature as the highest-scale key feature and value feature.

[0049] In step one, both the highest-scale key features and the highest-scale value features are derived from the highest-scale first point cloud features.

[0050] Step 2: Using the highest-scale first image features and first point cloud features, determine the highest-scale first attention weight and position offset.

[0051] In step two, the first image features and the first point cloud features at the highest scale are spliced ​​together, and the spliced ​​features are used to determine the first attention weight and position offset at the highest scale.

[0052] For example, the highest scale is the aforementioned B5 scale. The process of obtaining the first attention weight and position offset at the B5 scale can be represented by the following formula:

[0053] in, and These represent the B5-scale position offset and the first attention weight, respectively. The first image feature at the B5 scale. and the first point cloud features at the B5 scale There may be a misalignment in spatial location. First image feature used to correct B5 scale and the first point cloud features at the B5 scale Spatial position deviation.

[0054] Step 3: Using the highest-scale query features, key features, value features, first attention weights, and position offsets, determine the highest-scale second fusion features.

[0055] For example, the highest scale is the aforementioned B5 scale, and the second fusion feature obtained at the B5 scale can be represented by the following formula:

[0056] in, Q represents the second fusion feature at the B5 scale, while K and V represent the query feature, key feature, and value feature at the B5 scale, respectively.

[0057] The B5-scale positional offset includes the offset of each query position, and the B5-scale first attention weight includes the weight corresponding to each query position. The steps for determining the B5-scale second fusion feature include: sampling each query position in the B5-scale key and value features using the B5-scale positional offset (i.e., all positions of non-key and value features); determining the similarity between the B5-scale query features and the sampled B5-scale key features, and obtaining the B5-scale second attention weight; determining the product of the first and second attention weights to obtain the final B5-scale attention weight; and using the final B5-scale attention weight to perform a weighted summation of the sampled B5-scale value features to obtain the B5-scale second fusion feature. Detailed explanations of DCA processing can be found in known techniques and will not be elaborated upon here.

[0058] Step 4: Use the second fusion feature at the highest scale to determine the first fusion feature at the highest scale.

[0059] In step four, the sum of the second fusion feature at the highest scale and the first image feature at the highest scale can be determined as the first fusion feature at the highest scale. For example, if the highest scale is the aforementioned B5 scale, the first fusion feature at the B5 scale can be expressed by the following formula:

[0060] in, The first fusion feature at the B5 scale. This is the second fusion feature at the B5 scale. The first image feature is the one at the highest scale.

[0061] In step four, the highest-scale first fusion feature is determined by superimposing the second fusion feature at the highest scale and the first image feature at the highest scale. This allows the first image feature at the highest scale to retain the semantic information of the image while incorporating the depth information provided by the radar, thus improving the representational capability of the first fusion feature at the highest scale.

[0062] Alternatively, in other embodiments, a general cross-attention process can be used to fuse the first image feature and the first point cloud feature at the highest scale to obtain the first fused feature at the highest scale. The details of the cross-attention process will not be described in detail here.

[0063] In step S23, for the first scale among multiple scales, the first image feature and the first point cloud feature of the first scale are fused using reference information to obtain the first fused feature of the first scale. The reference information includes at least one of the first point cloud feature of the second scale and the first fused feature of the second scale. The first scale and the second scale are two adjacent scales among multiple scales, with the first scale being lower than the second scale.

[0064] For example, multiple scales include the aforementioned B3, B4, and B5 scales, where the first scale is the fourth scale and the second scale is the B5 scale, or the first scale is the B3 scale and the second scale is the B4 scale.

[0065] In one embodiment, before fusing the first image features and the first point cloud features at the first scale using reference information to obtain the first fused features at the first scale, the method further includes: adjusting the resolution corresponding to the reference information so that the resolution corresponding to the reference information is the same as the resolution corresponding to the first scale. Specifically, the reference information may be upsampled to increase the resolution corresponding to the reference information, so that the resolution corresponding to the reference information is the same as the resolution corresponding to the first scale.

[0066] For example, the first scale is B4, and the second scale is B5. The feature obtained after upsampling the first point cloud feature at scale B5 can be represented as follows: The upsampling of the first fused feature at the B5 scale can be expressed by the following formula:

[0067] in, Indicates to Upsampled to the resolution corresponding to the B4 scale. This is the first fusion feature at the B5 scale.

[0068] For example, the first scale is B3, and the second scale is B4. The feature obtained after upsampling the first point cloud feature at scale B4 can be represented as follows: The upsampling of the first fused feature at the B4 scale can be expressed by the following formula:

[0069] in, Indicates to Upsampled to the resolution corresponding to the B3 scale. This is the first fusion feature at the B4 scale.

[0070] In one embodiment, the reference information includes first point cloud features at a second scale. The first image features and the first point cloud features at the first scale are fused using the first point cloud features at the second scale and a general cross-attention process to obtain a first fused feature at the first scale. This embodiment includes the following steps: Step 1: Obtain the first image feature at the first scale as the query feature at the first scale, and obtain the first point cloud feature at the first scale and the first point cloud feature at the second scale together as the key feature and value feature at the first scale.

[0071] In step one, both the key features and the value features at the first scale include the first point cloud features at the first scale and the first point cloud features at the second scale.

[0072] Step 2: Using the query features, key features, and value features of the first scale, determine the second fusion feature of the first scale.

[0073] In step two, the similarity between the query features and key features at the first scale is determined to obtain the attention weight. The attention weight is then used to perform a weighted summation of the value features at the first scale to obtain the second fusion feature at the first scale.

[0074] Step 3: Use the second fusion feature of the first scale to determine the first fusion feature of the first scale.

[0075] In step three, the sum of the second fusion feature at the first scale and the first image feature at the first scale can be determined as the first fusion feature at the first scale, so as to enhance the representational ability of the first fusion feature at the first scale.

[0076] In this embodiment, by introducing high-scale first point cloud features into low-scale key features and low-scale value features, the low-scale fusion process can utilize high-scale radar point cloud information, thereby improving the accuracy of the determined low-scale second fusion features and thus improving the accuracy of the determined low-scale first fusion features.

[0077] In another embodiment, the reference information includes first point cloud features at a second scale and first fused features at a second scale. The first point cloud features at the second scale and the first fused features at the second scale are used, and DCA processing is combined to fuse the first image features and the first point cloud features at the first scale to obtain the first fused features at the first scale. This embodiment includes the following steps: Step 1: Obtain the first image feature at the first scale as the query feature at the first scale, and obtain the first point cloud feature at the first scale and the first point cloud feature at the second scale together as the key feature and value feature at the first scale.

[0078] In step one, both the key features and the value features at the first scale include the first point cloud features at the first scale and the first point cloud features at the second scale.

[0079] Step 2: Using the first image features at the first scale, the first point cloud features at the first scale, and the first fusion features at the second scale, determine the first attention weight and position offset at the first scale.

[0080] In step two, the first image features at the first scale, the first point cloud features at the first scale, and the first fusion features at the second scale are spliced ​​together, and the spliced ​​features are used to determine the first attention weight and position offset at the first scale.

[0081] For example, if the first scale is the aforementioned B4 scale, the process of obtaining the first attention weight and position offset at the B4 scale can be represented by the following formula:

[0082] in, and These represent the B4-scale position offset and the first attention weight, respectively. The first image feature is at the B4 scale. and the first point cloud features at the B4 scale There may be a misalignment in spatial location. First image feature used to correct B4 scale and the first point cloud features at the B4 scale Spatial position deviation.

[0083] For example, if the first scale is the aforementioned B3 scale, the process of obtaining the first attention weights and position offsets at the B3 scale can be represented by the following formula:

[0084] in, and These represent the B3-scale position offset and the first attention weight, respectively. The first image feature at the B3 scale. and the first point cloud features at the B3 scale There may be a misalignment in spatial location. First image feature used to correct B3 scale and the first point cloud features at the B3 scale Spatial position deviation.

[0085] Step 3: Using the query features, key features, value features, first attention weights, and position offsets of the first scale, determine the second fusion features of the first scale.

[0086] For example, if the first scale is the aforementioned B4 scale, the second fusion feature obtained at the B4 scale can be represented by the following formula:

[0087] in, The second fusion feature is the B4 scale feature, where Q, K, and V are the query feature, key feature, and value feature of the B4 scale, respectively.

[0088] For example, if the first scale is the aforementioned B3 scale, the second fusion feature obtained at the B3 scale can be represented by the following formula:

[0089] in, Q represents the second fusion feature at the B3 scale, while K and V represent the query feature, key feature, and value feature at the B3 scale, respectively.

[0090] The process of determining the second fusion feature at the first scale can refer to the process of determining the second fusion feature at the B5 scale, and will not be repeated here.

[0091] Step 4: Use the second fusion feature of the first scale to determine the first fusion feature of the first scale.

[0092] In step four, the sum of the second fusion feature at the first scale and the first image feature at the first scale can be determined as the first fusion feature at the first scale, so as to enhance the expressive power of the first fusion feature at the first scale.

[0093] For example, the first scale is the aforementioned B4 scale, and the first fusion feature of the B4 scale can be represented by the following formula:

[0094] in, This is the first fusion feature at the B4 scale. This is the second fusion feature at the B4 scale. The first image feature is at the B4 scale.

[0095] For example, if the first scale is the aforementioned B3 scale, the first fusion feature at the B3 scale can be represented by the following formula:

[0096] in, This is the first fusion feature at the B3 scale. This is the second fusion feature at the B3 scale. The first image feature is at the B3 scale.

[0097] In this embodiment, on the one hand, high-scale first point cloud features are introduced into the low-scale key features and low-scale value features, enabling the low-scale fusion process to utilize high-scale radar point cloud information, thereby improving the accuracy of the determined low-scale second fusion features and thus improving the accuracy of the determined low-scale first fusion features. On the other hand, the high-scale first fusion features (such as F_5^(fused_up_B4)) have already fused high-scale radar information, possessing stronger semantic and depth consistency and a stronger response to target category and existence. Introducing high-scale first fusion features into the determination of low-scale attention weights and position offsets can assist in predicting more accurate attention weights and position offsets, guiding the low-scale fusion process to favor regions deemed important by high-scale semantics.

[0098] S24: Fuse the first fusion features at each scale to obtain the target fusion features.

[0099] In one embodiment, fusing the first fusion features to obtain the target fusion feature includes the following steps: Step 1: Adjust the resolution of each first fusion feature to make the resolution of each first fusion feature the same.

[0100] In step one, the target resolution of the first fusion feature at the target scale can be used as the objective. Scales larger than the target scale are upsampled, and scales smaller than the target scale are downsampled, so that the resolution of each first fusion feature is adjusted to be the same as the resolution of the first fusion feature at the target scale.

[0101] For example, the resolution of the first fusion feature at the lowest scale among multiple scales is used as the target scale, and the first fusion features at the other scales are upsampled so that the resolution of the first fusion features at the other scales is the same as the resolution of the first fusion feature at the lowest scale.

[0102] For example, the resolution of the first fusion feature at the highest scale among multiple scales is used as the target scale, and the first fusion features at the other scales are downsampled so that the resolution of the first fusion features at the other scales is the same as the resolution of the first fusion feature at the highest scale.

[0103] Step 2: Perform feature splicing on each of the first fusion features to obtain spliced ​​features.

[0104] Step 3: Perform convolution processing on the spliced ​​features to obtain the target fused features.

[0105] Taking multiple scales, including the aforementioned B3, B4, and B5 scales, as an example, the obtained target fusion features can be represented by the following formula:

[0106] in, To achieve the goal of feature fusion, To Upsampled to the resolution corresponding to the B3 scale. To Upsampled to the resolution corresponding to the B3 scale. For feature splicing operations, This is for convolution processing.

[0107] S25: Use the target fusion features as the fusion result.

[0108] In this embodiment, the reference information is passed from high scale to low scale. The first point cloud feature and the first fusion feature at the high scale are passed to the low scale as reference information to guide the fusion process at the low scale. This allows the fusion process at the low scale to refer to the semantic information provided by the high scale, thereby improving the accuracy of the first fusion feature at the low scale. This improves the fusion effect of image data and radar point cloud data, and further improves the accuracy of the target detection result.

[0109] In this embodiment, the target fusion feature is obtained by fusing the first fusion features at each scale, and the target fusion feature is used as the fusion result. This can make full use of the detailed information and semantic abstraction between multi-scale features, enhance the expression of multi-scale features, and make the final output target fusion feature have strong semantics, rich details and high geometric accuracy. This further improves the fusion effect of image data and radar point cloud data, and can effectively reduce the false negative rate of small targets at a distance.

[0110] In this embodiment, image features can provide a semantic basis for the fusion result (such as target category, appearance, etc.), and radar point cloud features can provide a geometric basis for the fusion result (such as 3D position, depth, etc.). Cross-modal complementarity avoids the defects of single mode.

[0111] Please see Figure 3 , Figure 3 This is a schematic diagram of a framework of an embodiment of the fusion device provided in this application. In this embodiment, the fusion device 30 includes a feature acquisition module 31, a fusion module 32, and a result acquisition module 33.

[0112] The feature acquisition module 31 is used to acquire first image features at multiple scales corresponding to the image data, and to acquire first point cloud features at multiple scales corresponding to the radar point cloud data.

[0113] The fusion module 32 is used to fuse the first image features and the first point cloud features at each scale to obtain the first fused features corresponding to each scale. The first fused feature at the first scale is obtained by fusing the first image features and the first point cloud features at the first scale using reference information. The reference information includes at least one of the first point cloud features at the second scale and the first fused feature at the second scale. The first scale and the second scale are two adjacent scales among multiple scales, and the first scale is lower than the second scale.

[0114] The result acquisition module 33 is used to obtain the fusion result for the target detection task by utilizing each first fusion feature.

[0115] In one embodiment, the reference information includes first point cloud features at a second scale. The fusion module 32 is used to acquire first image features at a first scale as query features at the first scale, and to acquire first point cloud features at the first scale and first point cloud features at the second scale together as key features and value features at the first scale; using the query features, key features, and value features at the first scale, to determine second fusion features at the first scale; and using the second fusion features at the first scale, to determine first fusion features at the first scale.

[0116] In one embodiment, the reference information further includes a first fusion feature at a second scale. Before determining the second fusion feature at the first scale using the query feature, key feature, and value feature at the first scale, the fusion module 32 is further configured to determine a first attention weight and a position offset at the first scale using the first image feature at the first scale, the first point cloud feature at the first scale, and the first fusion feature at the second scale. Furthermore, the fusion module 32 is configured to determine the second fusion feature at the first scale using the query feature, key feature, value feature, first attention weight, and position offset at the first scale.

[0117] In one embodiment, the fusion module 32 is used to determine the sum of the second fusion feature at the first scale and the first image feature at the first scale as the first fusion feature at the first scale.

[0118] In one embodiment, before fusing the first image features and the first point cloud features at the first scale using reference information to obtain the first fused features at the first scale, the fusion module 32 is further used to adjust the resolution corresponding to the reference information so that the resolution corresponding to the reference information is the same as the resolution corresponding to the first scale.

[0119] In one embodiment, the result acquisition module 33 is used to fuse each of the first fusion features to obtain the target fusion feature; and the target fusion feature is used as the fusion result.

[0120] In one embodiment, the result acquisition module 33 is used to perform feature splicing on each of the first fusion features to obtain spliced ​​features; and to perform convolution processing on the spliced ​​features to obtain the target fusion features.

[0121] In one embodiment, before fusing the first fusion features to obtain the target fusion feature, the result acquisition module 33 is further used to adjust the resolution of each first fusion feature so that the resolution of each first fusion feature is the same.

[0122] It should be noted that the apparatus of this embodiment can perform the steps in the above method. For a detailed description of the relevant content, please refer to the method section above, which will not be repeated here.

[0123] Please see Figure 4 , Figure 4 This is a schematic diagram of a framework of an embodiment of the electronic device provided in this application. In this embodiment, the electronic device 40 includes a memory 41 and a processor 42.

[0124] Processor 42 can also be referred to as CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor, or processor 42 can be any conventional processor 42, etc.

[0125] The memory 41 in the electronic device 40 is used to store the program instructions required for the processor 42 to run.

[0126] The processor 42 is used to execute program instructions to implement the fusion method in this application.

[0127] Please seeFigure 5 , Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium provided in this application. The computer-readable storage medium 50 of this application embodiment stores program instructions 51, which, when executed, implement the fusion method provided in this application. The program instructions 51 can form a program file and be stored in the aforementioned computer-readable storage medium 50 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 50 includes various media capable of storing program code, such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

[0128] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0129] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A fusion method, characterized in that, The method includes: Acquire first image features at multiple scales corresponding to image data, and acquire first point cloud features at multiple scales corresponding to radar point cloud data; The first image features and the first point cloud features at each of the aforementioned scales are fused to obtain the first fused features corresponding to each of the aforementioned scales; wherein, the first fused feature at the first scale is obtained by fusing the first image features and the first point cloud features at the first scale using reference information, wherein the reference information includes at least one of the first point cloud features at the second scale and the first fused feature at the second scale, wherein the first scale and the second scale are two adjacent scales among the plurality of scales and the first scale is lower than the second scale. Using each of the first fusion features, a fusion result for the target detection task is obtained.

2. The method according to claim 1, characterized in that, The reference information includes the first point cloud features at the second scale. The first image features and the first point cloud features at the first scale are fused using the reference information to obtain the first fused feature at the first scale, including: The first image features at the first scale are obtained as query features at the first scale, and the first point cloud features at the first scale and the first point cloud features at the second scale are obtained together as key features and value features at the first scale. Using the query features, key features, and value features of the first scale, a second fusion feature of the first scale is determined; The first fusion feature at the first scale is determined using the second fusion feature at the first scale.

3. The method according to claim 2, characterized in that, The reference information also includes the first fusion feature at the second scale. Before determining the second fusion feature at the first scale using the query feature, the key feature, and the value feature at the first scale, the method further includes: Using the first image features at the first scale, the first point cloud features at the first scale, and the first fusion features at the second scale, the first attention weight and position offset at the first scale are determined. The step of determining the second fusion feature of the first scale using the query feature, the key feature, and the value feature of the first scale includes: The second fusion feature of the first scale is determined by using the query feature, the key feature, the value feature, the first attention weight, and the position offset of the first scale.

4. The method according to claim 2, characterized in that, The step of determining the first fusion feature at the first scale using the second fusion feature at the first scale includes: The sum of the second fusion feature at the first scale and the first image feature at the first scale is determined as the first fusion feature at the first scale.

5. The method according to claim 1, characterized in that, Before fusing the first image features and the first point cloud features at the first scale using the reference information to obtain the first fused features at the first scale, the method further includes: Adjust the resolution corresponding to the reference information so that the resolution corresponding to the reference information is the same as the resolution corresponding to the first scale.

6. The method according to claim 1, characterized in that, The step of obtaining the fusion result for the target detection task by utilizing each of the first fusion features includes: By fusing the first fusion features, the target fusion feature is obtained; The target fusion feature is used as the fusion result.

7. The method according to claim 6, characterized in that, The fusion of each of the first fusion features to obtain the target fusion feature includes: The first fused features are concatenated to obtain the concatenated features; The spliced ​​features are subjected to convolution processing to obtain the target fused features.

8. The method according to claim 6, characterized in that, Before fusing each of the first fusion features to obtain the target fusion feature, the method further includes: Adjust the resolution of each of the first fused features so that the resolution of each of the first fused features is the same.

9. An electronic device, characterized in that, Including interconnected memory and processor, The memory stores program instructions; The processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed by a processor to implement the method of any one of claims 1-8.

Images