Data fusion method

Abstract

The invention relates to the technical field of multi-modal data fusion, in particular to a data fusion method and device, a computer program product and a vehicle. To-be-fused data at least comprises image data and radar data, and the data fusion method comprises the following steps: extracting shallow spatial features of the image data; performing shallow coding on the radar data to obtain radar features; and inputting the shallow spatial features and the radar features into a fusion network to obtain fusion features output by the fusion network.

Description

Technical Field

[0001] This disclosure relates to the field of multimodal data fusion technology, and in particular to a data fusion method, apparatus, computer program product, and vehicle. Background Technology

[0002] In related technologies, multimodal data fusion relies on large encoders and complex fusion mechanisms, which consumes a lot of computing resources and incurs high overhead, making it difficult to meet the requirements of low latency and low cost in some scenarios. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a data fusion method, apparatus, computer program product, and vehicle that can solve the above-mentioned problems.

[0004] According to a first aspect of the present disclosure, a data fusion method is provided, wherein the data to be fused includes at least image data and radar data, the method comprising: extracting shallow spatial features from the image data; performing shallow encoding on the radar data to obtain radar features; and inputting the shallow spatial features and the radar features into a fusion network to obtain fused features output by the fusion network.

[0005] According to a second aspect of the present disclosure, a data fusion apparatus is provided, wherein the data to be fused includes at least image data and radar data. The apparatus includes: an extraction unit configured to extract shallow spatial features from the image data; an encoding unit configured to perform shallow encoding on the radar data to obtain radar features; and a fusion unit configured to input the shallow spatial features and the radar features into a fusion network to obtain fused features output by the fusion network.

[0006] According to a third aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in the first aspect.

[0007] According to a fourth aspect of the present disclosure, a vehicle is provided, wherein an electronic control unit on the vehicle is configured to perform the method as described in the first aspect.

[0008] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: The data fusion method proposed in this disclosure can fuse at least image data and radar data. It extracts shallow spatial features from the image data, then performs shallow encoding on the radar data to obtain radar features. A trained fusion network is then used to obtain fused features based on the input shallow spatial features and radar features. In this disclosure, shallow spatial features are extracted from the image data, thus requiring a relatively lightweight feature extraction network, containing only 3-5 convolutional layers, resulting in less computational resources. In contrast, convolutional networks used for image feature extraction in related technologies typically contain over a hundred layers, requiring significantly more computational resources. Secondly, this disclosure obtains radar features based on shallow encoding, which requires a smaller network size compared to related technologies, further reducing computational resource demands. Finally, based on the fusion network used in this disclosure, the two modalities can be fused using only the shallow spatial features of the image data and the radar features of the radar data, enabling modal alignment even when shallow spatial features and shallow encoding are employed.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0011] Figure 1 This is a schematic flowchart illustrating a data fusion method according to an exemplary embodiment of the present disclosure.

[0012] Figure 2 This is a schematic flowchart illustrating a data fusion method according to an exemplary embodiment of the present disclosure.

[0013] Figure 3 This disclosure is a schematic diagram illustrating a training process for a fusion network according to an exemplary embodiment.

[0014] Figure 4 This is a block diagram illustrating a data fusion apparatus according to an exemplary embodiment of the present disclosure.

[0015] Figure 5 This is a schematic block diagram illustrating a data fusion apparatus according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0016] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0017] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0018] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0019] Currently, various types of sensors can collect multimodal sensor data, such as image data, radar data, and acceleration data. To achieve a more comprehensive perception of the surrounding environment or target objects, different multimodal sensor data can be fused to obtain more accurate fused data. Multimodal data fusion has applications in many fields, such as vehicle perception technology and robot recognition technology.

[0020] The relevant technologies rely on large encoders and complex fusion mechanisms. The learning networks involved may contain hundreds of convolutional layers, resulting in large computational overhead and making it difficult to meet the requirements of low latency and low cost for multimodal data fusion in multiple scenarios.

[0021] Taking the field of vehicle technology as an example, the computing power and power consumption of in-vehicle edge devices (ECUs, Electronic Control Units) are limited, and the computational complexity of related technologies in the feature extraction stage and fusion network is still too high for ECUs with low computing resources, making it difficult to perform multimodal data fusion in devices with limited computing resources.

[0022] To address the aforementioned technical issues, this disclosure proposes a data fusion method.

[0023] Figure 1 This is a schematic flowchart illustrating a data fusion method according to an embodiment of the present disclosure.

[0024] In some embodiments, the data to be fused includes at least image data and radar data.

[0025] The processing of image data and radar data requires relatively large resources. This disclosure mainly focuses on optimizing the fusion of image data and radar data, thereby reducing the demand on the computing resources of the equipment.

[0026] For example, the data to be fused may also include acceleration information, audio information, etc.

[0027] like Figure 1 As shown, data fusion methods include: In step S101, shallow spatial features of the image data are extracted, and the shallow spatial features include the geometric features of the image data; In step S102, the radar data is shallowly encoded to obtain radar features, which are used to characterize the geometric features and spatial positional relationships of the radar data. In step S103, the shallow spatial features and the radar features are input into a fusion network to obtain the fused features output by the fusion network.

[0028] In some embodiments, multimodal data from different sensors can be acquired.

[0029] Multiple different sensors send the collected data to the processing unit, which then executes the data fusion method proposed in this disclosure to fuse the multimodal data.

[0030] For example, image data can be acquired from an image acquisition device, or radar data can be acquired from a radar. Radar data may include, but is not limited to, point cloud data, which contains information such as the position and velocity of targets detected by radar.

[0031] In some embodiments, the processing unit used to perform the data fusion method proposed in this disclosure has limited computing power.

[0032] For example, a vehicle edge computing device (ECU) may have limited computing power (e.g., only 5 TOPS (5 TeraOperations Per Second)) and cannot employ the data fusion methods in related technologies. This disclosure requires data fusion to be achieved based on its limited computing resources.

[0033] In some embodiments, shallow spatial features of the image data are extracted.

[0034] Due to the limited computational power of the processing unit, it cannot support large-scale CNNs (Convolutional Neural Networks), such as ResNet. In the examples disclosed herein, image data can be processed using a lightweight first network.

[0035] For example, the first network could be a small CNN containing only 3-5 convolutional layers, with a total number of parameters of less than 0.5M.

[0036] By processing image data I through this first network, shallow spatial features of the image data can be extracted. The output dimension is .

[0037] Shallow spatial features encompass the geometric features of the image data. These features preserve the geometric structure of the image data and can extract basic geometric elements such as edges and corners. This allows for the summarization of key image features with minimal computational resource consumption. For example, in vehicle driving scenarios, shallow spatial features are beneficial for distinguishing lane line outlines and obstacle boundaries. Furthermore, changes in shallow spatial features across consecutive frames can directly estimate object motion vectors, thereby enhancing motion perception.

[0038] Since fewer convolutional layers are required to extract shallow spatial features, the computational resource requirements are also lower, which can be controlled below 0.1 TOPS. This allows for the simultaneous parallel processing of more image data, ensuring real-time performance and minimizing latency.

[0039] In some embodiments, the radar data is shallowly encoded to obtain radar features, which are used to characterize the geometric features and spatial relationships of the radar data.

[0040] For example, radar data can include radar target point clouds with coefficients. .

[0041] A small second network can be used to perform shallow encoding on radar data to obtain radar features. .

[0042] For example, the second network could be a 3-layer MLP (Multi-Layer Perception) and a possible point set processing layer (e.g., the PointNet base layer), with a total number of parameters of less than 0.2M.

[0043] Shallow coding is used to obtain radar features, which can effectively highlight target features in radar data, enhance motion features, and help isolate noise.

[0044] The radar features obtained by shallow coding retain the underlying geometric structure and physical properties of the radar data, maintain high spatial resolution, have low semantic information, but clear spatial positional relationships, and have strong real-time performance based on the strong spatial details in the radar features.

[0045] In some embodiments, the shallow spatial features and the radar features are input into a fusion network to obtain the fused features output by the fusion network.

[0046] The extracted shallow spatial features With radar characteristics Input a lightweight fusion network, which fuses shallow spatial features and radar features to output fused features.

[0047] For example, a fusion network can be a small network containing two layers of cross-attention and two layers of MLP, such as the Transformer, with a total number of parameters of approximately 2.5M.

[0048] This fusion network is pre-trained and can effectively fuse shallow spatial features and radar features, with good alignment results.

[0049] This disclosure uses a lightweight network to extract shallow spatial features from image data, performs shallow encoding on radar data to obtain radar features, and then uses a fusion network to obtain fused features based on the shallow spatial features and the radar features obtained from the shallow encoding. Since lightweight networks are used in both image and radar data feature extraction, and only shallow features are extracted, the method of this disclosure requires relatively few computational resources from the processing unit, and can be implemented even on resource-constrained devices. Furthermore, the fusion network pre-trained based on the shallow spatial features and radar features obtained from the shallow encoding, as disclosed in this disclosure, can fuse data based on shallow features and exhibits good alignment performance.

[0050] Figure 2 This is a schematic flowchart illustrating a data fusion method according to an embodiment of the present disclosure.

[0051] like Figure 2 As shown, this disclosure can process image data and radar data, input the processed data into a pre-trained fusion network, and then output fused features based on the fusion network.

[0052] The processing of image data may include extracting shallow spatial features and preprocessing (e.g., projection), while the processing of radar data may include shallow coding performed by a lightweight encoder.

[0053] In some embodiments, the method further includes: projecting the shallow spatial features to determine projected features; and inputting the shallow spatial features and the radar features into a fusion network, which includes: inputting the projected features and the radar features into the fusion network.

[0054] Due to shallow spatial characteristics yes Dimensions, and radar characteristics yes Dimensionality, and needs to be adapted to the input of the fusion network, so it is possible to first... Shallow spatial features of dimensionality Preprocessing is performed.

[0055] For example, shallow spatial features can be used Flatten is The dimension is then reduced to 1 through linear layer projection (Proj). Dimensions, to determine projection features .

[0056] Input-based projection features and radar characteristics Fusion networks can learn cross-modal interactions between two features and output the final fused features. .

[0057] In some embodiments, the image data includes low-resolution images with a resolution below a first resolution threshold; the radar data includes point cloud data.

[0058] The higher the resolution of image data, the greater the computational resources required to process it. Therefore, low-resolution images with a resolution below a first resolution threshold can be selected. (For example This allows for a better reduction in the demand for computing resources.

[0059] Furthermore, since the processing of image data in this disclosure involves extracting shallow spatial features through a network, low-resolution or high-resolution images do not affect the final processing effect. Even when using low-resolution images, good data fusion results can still be achieved after extracting shallow spatial features.

[0060] Radar data can include point cloud data, such as sparse target point clouds. Similar to image data, the denser the point cloud data, the greater the amount of information, but the more computational resources are required for processing. Therefore, using sparse target point clouds can reduce the demand for computational resources. Sparse target point clouds also contain the position and velocity information of the target objects detected by the radar sensor, and the radar features obtained from the shallow encoding can also be fused with the shallow spatial features of image data.

[0061] In some embodiments, the method further includes: acquiring a high-resolution image collected by a sensor; and compressing the high-resolution image into the low-resolution image.

[0062] Some sensors acquire image data with high resolution. Based on this embodiment, high-resolution images can be compressed into low-resolution images, and then multimodal data fusion can be performed using the data fusion method proposed in this disclosure. Compressing high-resolution images into low-resolution images eliminates the need for additional low-resolution sensors for the data fusion method of this disclosure, allowing the reuse of existing image sensors. Furthermore, it allows the reuse of high-resolution images acquired by sensors, eliminating the need for additional image data specifically acquired for the data fusion method of this disclosure. For example, the acquired high-resolution images can be used for image display, and simultaneously compressed for the data fusion proposed in this disclosure.

[0063] In some embodiments, the method further includes: acquiring point cloud data collected by a radar sensor, wherein the point cloud data is a dense point cloud; and filtering the dense point cloud to determine sparse point cloud data.

[0064] Radar acquires relatively dense point cloud data to handle other sensor tasks. However, to reduce the computational resource requirements for data fusion, this disclosure filters the acquired dense point cloud data to extract a few key point cloud data points, thereby obtaining sparse point cloud data for use in the data fusion method of this disclosure.

[0065] In some embodiments, the fusion features are input into a downstream target task.

[0066] Fusion characteristics of the output of the fusion network This fusion feature can be used as It is represented in the form of a 3D vector that integrates image data and radar data. Its compactness makes it more suitable as input for downstream target tasks, which can be used for various downstream target tasks that require fast response and have limited computing resources.

[0067] For example, when the data fusion method proposed in this disclosure is applied to a vehicle, the downstream target task can be a downstream vehicle application, including but not limited to: a real-time obstacle warning system and low-power scenario classification.

[0068] For example, fusion features can be used The data is input into a small collision risk assessment model, such as a small MLP, to obtain the output collision risk level or predicted TTC (Time-to-Collision), thereby realizing a real-time obstacle warning function.

[0069] For example, fusion features can be used The input is fed into a simple classifier (e.g., a 2-layer MLP) to determine the current scene type of the vehicle, thus achieving low-power scene classification. The scene types of the vehicle are in include, but are not limited to: urban driving scene, highway driving scene, and rural driving scene.

[0070] The embodiments of this disclosure can obtain matching fused features based solely on shallow spatial features of image data and radar features obtained through shallow encoding of radar data, relying on the fusion network specifically trained by this disclosure.

[0071] The fusion network will be further described below through examples.

[0072] Figure 3 This is a schematic diagram illustrating a training process for a fusion network according to an embodiment of the present disclosure.

[0073] like Figure 3 As shown, image feature distribution can be obtained based on shallow spatial features of image data, and radar feature distribution can be obtained based on radar features. Then, the distribution difference between the two can be determined based on the image feature distribution and the radar feature distribution, and finally, a loss function for this difference can be determined. This determined loss function is used to train the fusion network.

[0074] Since the loss function includes the difference between the image feature distribution and the radar feature distribution, training the fusion network based on this loss function can optimize the fusion network to address the differences between the image feature distribution and the radar feature distribution during the fusion process. This ensures that the image feature distribution can be aligned with the radar feature distribution, enabling the data fusion method proposed in this disclosure to achieve good multimodal data alignment results based solely on radar features obtained from shallow spatial features and shallow coding. This allows for the generation of fused features from multimodal data fusion.

[0075] In some embodiments, the fusion network is optimized during the training phase by: determining the maximum mean difference between the distribution of the shallow spatial features and the distribution of the radar features; and optimizing the fusion network based on a first loss function determined by the maximum mean difference.

[0076] This embodiment is applied to the offline training phase of the fusion network. When training the model (fusion network) parameters, a distribution distance loss between multimodal data can be introduced. (First loss function). Distance loss based on this distribution. It can be based on shallow spatial features (e.g., those obtained after preprocessing) Image features after projection of dimensionality The distribution of radar characteristics (e.g.) Radar characteristics of dimensional The maximum mean discrepancy (MMD) between the distributions of ) is determined.

[0077] It should be noted that, unlike related technologies that use deep networks to extract features from image and radar data, the embodiments of this disclosure use lightweight networks to reduce resource consumption. The network extracts shallow spatial features from the image data and shallowly encoded radar features from the radar data. Since these two types of features contain a small amount of relatively simple information, it is difficult to achieve good alignment based on the fusion networks in related technologies, thus making it difficult to fuse the data.

[0078] In this embodiment, the fusion network is specifically optimized to align and fuse shallow spatial features with radar features extracted through shallow coding. Since image data and radar data collected from the same measurement object can contain similar features and have similar distributions, this embodiment first determines the maximum mean difference (MMD) between the shallow spatial features and radar features, and then trains the fusion network based on the MMD loss. This allows the trained fusion network to align the shallow spatial features and radar features, thereby achieving fusion.

[0079] In some embodiments, the method further includes: inputting the fusion features into a downstream target task; optimizing the fusion network based on a first loss function determined according to the maximum mean difference includes: optimizing the fusion network based on the first loss function and a second loss function; wherein the second loss function is the loss function of the downstream target task, and the downstream target task is implemented based on the fusion features output by the fusion network.

[0080] The fused features output by the fusion network are used as input for downstream target tasks. Therefore, the optimization of the fusion network can also be performed on downstream target tasks.

[0081] For example, the first loss function could be the intermodal distribution distance loss. The second loss function is the loss function for the downstream target task, which can be used... express.

[0082] The total loss function can be determined based on the first loss function and the second loss function, and then the fusion network can be optimized based on the total loss function.

[0083] The total loss function can be expressed as follows:

[0084] in, This is the total loss function; This is the loss function for the downstream target task.

[0085] The loss function for the downstream target task can be flexibly set based on the specific downstream target task. For example, if the output fused features need to be used for scene classification, that is, if the downstream target task is scene classification, then the loss function for the downstream target task can be cross-entropy loss; if the output fused features need to be used for collision risk prediction, that is, if the downstream target task is collision risk regression, then the loss function for the downstream target task can be mean squared error (MSE) loss.

[0086] The weight for the MMD loss can be, for example, 0.15.

[0087] The function used to measure the difference between two probability distributions is used in embodiments of this disclosure to determine the difference between an image feature distribution and a radar feature distribution, i.e., to determine the MMD loss. The MMD can be measured by calculating the squared distance between the mean embeddings of the two feature distributions in the reproducing kernel Hilbert space (RKHS), which in embodiments of this disclosure is used to characterize whether the image features and radar features are aligned.

[0088] In some embodiments, determining the maximum mean difference between the distribution of the shallow spatial features and the distribution of the radar features includes: determining a first empirical probability distribution of the shallow spatial features and a second empirical probability distribution of the radar features, respectively; and determining the maximum mean difference based on the first empirical probability distribution and the second empirical probability distribution.

[0089] Please refer to the expression for the total loss function above. The function is used to represent the empirical probability distribution of the sample feature vector. In embodiments of this disclosure, the function is used based on the projected image features (shallow spatial features) of the input. ( (dimensional) and shallow-coded radar features ( (Dimension), determine the corresponding distribution.

[0090] Since the total loss function includes the first loss function (MMD loss), the multimodal features learned by the fusion network have similar statistical distribution characteristics in the shared embedding space after projection. This is beneficial for the alignment of multimodal data features, thereby improving the fusion effect.

[0091] Based on one embodiment, the first network used to extract shallow spatial features of image data in this disclosure can be a shallow CNN with fewer than 0.5M parameters, a lightweight encoder radar MLP used to perform shallow encoding of radar data with fewer than 0.2M parameters, and the number of parameters of the trained fusion network is about 2.5M. This makes the total number of inference parameters required by the data fusion method proposed in this disclosure about 3.2M, with low computational resource requirements. It can run effectively even on devices with limited computational resources (e.g., 5 TOPS), and has strong real-time performance, enabling it to quickly output fused features.

[0092] In some embodiments, the fusion network can also be trained by employing knowledge distillation techniques to allow the fusion network to inherit from a teacher network that consumes more resources than the fusion network itself, the teacher network being used to align and fuse the image data with the radar data.

[0093] Unlike training fusion networks based on the MMD loss function, this disclosure also proposes another method for training fusion networks. Knowledge distillation can be employed, enabling the fusion network, acting as a "student" model, to learn and inherit the capabilities of a "teacher" model, which has excellent data fusion performance but consumes significant resources, while maintaining extremely low resource consumption.

[0094] By learning and inheriting from the teacher network (“teacher” model), unnecessary parameters can be optimized, thereby reducing the size of the fusion network. This allows the fusion network to have a better fusion alignment effect when facing shallow spatial features and radar features, thus reducing resource consumption while achieving the alignment and fusion of multimodal data.

[0095] Corresponding to the embodiments of the data fusion method of this disclosure, this disclosure also provides embodiments of a corresponding data fusion apparatus, wherein the data to be fused includes at least image data and radar data.

[0096] Please see Figure 4 , Figure 4 This is a block diagram of a data fusion apparatus according to one embodiment of this disclosure. Figure 4 As shown, the data fusion device includes: Extraction unit 410 is configured to extract shallow spatial features of the image data, the shallow spatial features including the geometric features of the image data; Encoding unit 420 is configured to perform shallow encoding on the radar data to obtain radar features, the radar features being used to characterize the geometric features and spatial positional relationships of the radar data. The fusion unit 430 is configured to input the shallow spatial features and the radar features into a fusion network to obtain the fused features output by the fusion network.

[0097] In some embodiments, the apparatus further includes: a projection unit configured to project the shallow spatial features to determine projection features; and a fusion unit specifically configured to input the projection features and the radar features into a fusion network.

[0098] In some embodiments, the image data includes low-resolution images with a resolution below a first resolution threshold; the radar data includes point cloud data.

[0099] In some embodiments, the apparatus further includes: a compression unit configured to acquire a high-resolution image from a sensor; and to compress the high-resolution image into the low-resolution image.

[0100] In some embodiments, the fusion network is optimized during the training phase by means of: a fusion network training apparatus configured to determine the maximum mean difference between the distribution of the shallow spatial features and the distribution of the radar features; and to optimize the fusion network based on a first loss function determined by the maximum mean difference.

[0101] In some embodiments, the apparatus further includes: an application unit configured to input the fusion features into a downstream target task; the optimization of the fusion network based on a first loss function determined according to the maximum mean difference includes: optimizing the fusion network based on the first loss function and a second loss function; wherein the second loss function is a loss function of the downstream target task, and the downstream target task is implemented based on the fusion features output by the fusion network.

[0102] In some embodiments, determining the maximum mean difference between the distribution of the shallow spatial features and the distribution of the radar features includes: determining a first empirical probability distribution of the shallow spatial features and a second empirical probability distribution of the radar features, respectively; and determining the maximum mean difference based on the first empirical probability distribution and the second empirical probability distribution.

[0103] In some embodiments, the fusion network is trained by a fusion network training apparatus configured to employ knowledge distillation techniques to enable the fusion network to inherit from a teacher network that has higher resource consumption than the fusion network, the teacher network being used to align and fuse the image data with the radar data.

[0104] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0105] Embodiments of this disclosure also provide an electronic device, including: a processor and a memory; the memory for storing a computer program; and the processor for executing a data fusion method as described in any of the above embodiments by invoking the computer program.

[0106] Embodiments of this disclosure also propose a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the data fusion method as described in any of the above embodiments.

[0107] Embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0108] Figure 5 This is a schematic block diagram illustrating a data fusion apparatus 500 according to embodiments of the present disclosure. For example, apparatus 500 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0109] Reference Figure 5 The device 500 may include one or more of the following components: processing component 502, memory 504, power supply component 506, multimedia component 508, audio component 510, input / output (I / O) interface 512, sensor component 514, and communication component 516.

[0110] Processing component 502 typically controls the overall operation of device 500, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the data fusion method described above. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0111] Memory 504 is configured to store various types of data to support the operation of device 500. Examples of such data include instructions for any application or method operating on device 500, contact data, phonebook data, messages, pictures, videos, etc. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0112] Power supply component 506 provides power to various components of device 500. Power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 500.

[0113] Multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0114] Audio component 510 is configured to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) configured to receive external audio signals when device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0115] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0116] Sensor assembly 514 includes one or more sensors for providing status assessments of various aspects of device 500. For example, sensor assembly 514 may detect the on / off state of device 500, the relative positioning of components such as the display and keypad of device 500, changes in position of device 500 or a component of device 500, the presence or absence of user contact with device 500, orientation or acceleration / deceleration of device 500, and temperature changes of device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0117] Communication component 516 is configured to facilitate wired or wireless communication between device 500 and other devices. Device 500 can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G LTE, 5G NR, or combinations thereof. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0118] In an exemplary embodiment, the apparatus 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the data fusion method described above.

[0119] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by a processor 520 of the device 500 to complete the data fusion method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0120] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0121] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0122] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0123] The methods and apparatus provided in the embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A data fusion method, characterized in that, The data to be fused includes at least image data and radar data, and the method includes: Extract shallow spatial features from the image data, wherein the shallow spatial features include the geometric features of the image data; The radar data is shallowly encoded to obtain radar features, which are used to characterize the geometric features and spatial relationships of the radar data. The shallow spatial features and the radar features are input into a fusion network to obtain the fused features output by the fusion network.

2. The method according to claim 1, characterized in that, The method further includes: projecting the shallow spatial features to determine the projected features; The step of inputting the shallow spatial features and the radar features into the fusion network includes: inputting the projection features and the radar features into the fusion network.

3. The method according to claim 1, characterized in that, The image data includes low-resolution images with a resolution lower than a first resolution threshold; the radar data includes point cloud data.

4. The method according to claim 3, characterized in that, The method further includes: Acquire high-resolution images from the sensor; The high-resolution image is compressed into the low-resolution image.

5. The method according to any one of claims 1-4, characterized in that, The fusion network is optimized during the training phase using the following methods: Determine the maximum mean difference between the distribution of the shallow spatial features and the distribution of the radar features; The fusion network is optimized based on a first loss function determined by the maximum mean difference.

6. The method according to claim 5, characterized in that, The method further includes: inputting the fused features into a downstream target task; The optimization of the fusion network based on the first loss function determined according to the maximum mean difference includes: optimizing the fusion network based on the first loss function and the second loss function; wherein the second loss function is the loss function of the downstream target task, and the downstream target task is implemented based on the fusion features output by the fusion network.

7. The method according to claim 5, characterized in that, Determining the maximum mean difference between the distribution of the shallow spatial features and the distribution of the radar features includes: Determine the first empirical probability distribution of the shallow spatial features and the second empirical probability distribution of the radar features, respectively. The maximum mean difference is determined based on the first empirical probability distribution and the second empirical probability distribution.

8. The method according to any one of claims 1-4, characterized in that, The fusion network is trained using the following method: The knowledge distillation technique is used to enable the fusion network to inherit from the teacher network, which has higher resource consumption than the fusion network. The teacher network is used to realize the alignment and fusion of the image data and the radar data.

9. A data fusion device, characterized in that, The data to be fused includes at least image data and radar data, and the device includes: An extraction unit is configured to extract shallow spatial features of the image data, the shallow spatial features including the geometric features of the image data; The encoding unit is configured to perform shallow encoding on the radar data to obtain radar features, which are used to characterize the geometric features and spatial relationships of the radar data. The fusion unit is configured to input the shallow spatial features and the radar features into a fusion network to obtain the fused features output by the fusion network.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

11. A vehicle, characterized in that, The electronic control unit on the vehicle is used to perform the method described in any one of claims 1-8.

Images