Image fusion method, device, vehicle and storage medium

Abstract

The present disclosure relates to a method, device, vehicle and storage medium for image fusion, and relates to the field of multimedia technology. The method comprises: acquiring images acquired by multiple image acquisition devices. Acquiring the image features of each image. Using the image features of each image as the input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model. The image fusion model is used to splice multiple image features into a target feature vector, encode the target feature vector to obtain an encoded encoding vector, and decode the encoding vector to obtain a target image. The present disclosure encodes and decodes multiple images acquired by multiple image acquisition devices through an image fusion model to obtain a target image after the multiple images are fused, which can improve the accuracy of image fusion.

Description

Technical Field

[0001] The present disclosure relates to the field of multimedia technology, and in particular to an image fusion method, device, vehicle and storage medium. Background Art

[0002] With the continuous development of society, self-driving cars are increasingly used in people's lives. In the perception task of self-driving cars, the vehicle needs to perceive 360-degree information around the vehicle body. This information often comes from multiple image acquisition devices (such as cameras and radars), and the acquisition areas of multiple image acquisition devices may overlap, resulting in the same object appearing in multiple images. In related technologies, it is impossible to accurately process objects that appear repeatedly in multiple images, resulting in the vehicle being unable to accurately obtain information around the vehicle body. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides an image fusion method, device, vehicle and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for image fusion is provided, the method comprising:

[0005] Acquire images acquired by multiple image acquisition devices;

[0006] Obtain image features for each image;

[0007] The image features of each image are used as input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model. The image fusion model is used to splice multiple image features into a target feature vector, encode the target feature vector to obtain an encoded encoding vector, and decode the encoding vector to obtain the target image.

[0008] Optionally, the image fusion model includes an encoding module and a decoding module, and the encoding module is used to:

[0009] Encoding the target feature vector according to a first encoding method to obtain a first encoding vector;

[0010] Encoding the target feature vector according to a second encoding method to obtain a second encoding vector;

[0011] The decoding module is used for:

[0012] The first coding vector and the second coding vector are decoded to obtain the target image.

[0013] Optionally, encoding the target feature vector according to a first encoding mode to obtain a first encoding vector includes:

[0014] Performing a dimension transformation on the target feature vector to obtain a first transformed feature vector;

[0015] Performing position encoding on the first transformed feature vector to obtain a first position vector;

[0016] The sum of the first transformed feature vector and the first position vector is used as the first encoding vector.

[0017] Optionally, performing a dimension transformation on the target feature vector to obtain a first transformed feature vector includes:

[0018] Flattening the height and width of the target feature vector to obtain a first intermediate feature vector;

[0019] When the number of dimensions in the first intermediate feature vector is not equal to the preset number of dimensions, a linear transformation is performed on the first intermediate feature vector to obtain the first transformed feature vector.

[0020] Optionally, the performing dimension transformation on the target feature vector to obtain a first transformed feature vector further includes:

[0021] When the number of dimensions in the first intermediate feature vector is equal to the preset number of dimensions, the first intermediate feature vector is used as the first transformed feature vector.

[0022] Optionally, encoding the target feature vector according to a second encoding method to obtain a second encoding vector includes:

[0023] Performing a dimension transformation on the target feature vector to obtain a second transformed feature vector;

[0024] Performing position encoding on the second transformed feature vector to obtain a second position vector;

[0025] The sum of the second transformed feature vector, the second position vector and a preset query vector is used as the second encoding vector.

[0026] Optionally, performing a dimension transformation on the target feature vector to obtain a second transformed feature vector includes:

[0027] Performing pooling processing on the target feature vector to obtain a second intermediate feature vector;

[0028] When the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is not equal to the number of dimensions in the query vector, a linear transformation is performed on the second intermediate feature vector to obtain the second transformed feature vector.

[0029] Optionally, the performing dimension transformation on the target feature vector to obtain a second transformed feature vector further includes:

[0030] When the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is equal to the number of dimensions in the query vector, the second intermediate feature vector is used as the second transformed feature vector.

[0031] According to a second aspect of an embodiment of the present disclosure, a device for image fusion is provided, the device comprising:

[0032] A first acquisition module is configured to acquire images acquired by multiple image acquisition devices;

[0033] A second acquisition module is configured to acquire image features of each image;

[0034] A fusion module is used to use the image features of each image as the input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model. The image fusion model is used to splice multiple image features into a target feature vector, encode the target feature vector to obtain an encoded encoding vector, and decode the encoding vector to obtain the target image.

[0035] Optionally, the image fusion model includes an encoding module and a decoding module, and the encoding module is used to:

[0036] Encoding the target feature vector according to a first encoding method to obtain a first encoding vector;

[0037] Encoding the target feature vector according to a second encoding method to obtain a second encoding vector;

[0038] The decoding module is used for:

[0039] The first coding vector and the second coding vector are decoded to obtain the target image.

[0040] Optionally, the encoding module is used to:

[0041] Performing a dimension transformation on the target feature vector to obtain a first transformed feature vector;

[0042] Performing position encoding on the first transformed feature vector to obtain a first position vector;

[0043] The sum of the first transformed feature vector and the first position vector is used as the first encoding vector.

[0044] Optionally, the encoding module is used to:

[0045] Flattening the height and width of the target feature vector to obtain a first intermediate feature vector;

[0046] When the number of dimensions in the first intermediate feature vector is not equal to the preset number of dimensions, a linear transformation is performed on the first intermediate feature vector to obtain the first transformed feature vector.

[0047] Optionally, the encoding module is used to:

[0048] When the number of dimensions in the first intermediate feature vector is equal to the number of dimensions in the preset query vector, the first intermediate feature vector is used as the first transformed feature vector.

[0049] Optionally, the encoding module is used to:

[0050] Performing a dimension transformation on the target feature vector to obtain a second transformed feature vector;

[0051] Performing position encoding on the second transformed feature vector to obtain a second position vector;

[0052] The sum of the second transformed feature vector, the second position vector and a preset query vector is used as the second encoding vector.

[0053] Optionally, the encoding module is used to:

[0054] Performing pooling processing on the target feature vector to obtain a second intermediate feature vector;

[0055] When the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is not equal to the number of dimensions in the query vector, a linear transformation is performed on the second intermediate feature vector to obtain the second transformed feature vector.

[0056] Optionally, the encoding module is used to:

[0057] When the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is equal to the number of dimensions in the query vector, the second intermediate feature vector is used as the second transformed feature vector.

[0058] According to a third aspect of an embodiment of the present disclosure, a vehicle is provided, comprising:

[0059] processor;

[0060] a memory for storing processor-executable instructions;

[0061] Wherein, the processor is configured to:

[0062] Acquire images acquired by multiple image acquisition devices;

[0063] Obtain image features for each image;

[0064] The image features of each image are used as input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model. The image fusion model is used to splice multiple image features into a target feature vector, encode the target feature vector to obtain an encoded encoding vector, and decode the encoding vector to obtain the target image.

[0065] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of any one of the methods described in the first aspect of the present disclosure are implemented.

[0066] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0067] The present disclosure first obtains images collected by multiple image acquisition devices, and obtains the image features of each image. Then, the image features of each image are used as the input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model, wherein the image fusion model is used to splice multiple image features into a target feature vector, encode the target feature vector, obtain an encoded encoding vector, and decode the encoding vector to obtain a target image. The present disclosure encodes and decodes multiple images collected by multiple image acquisition devices through an image fusion model to obtain a target image after the fusion of multiple images, which can improve the accuracy of image fusion.

[0068] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0070] Figure 1 is a flow chart of an image fusion method according to an exemplary embodiment;

[0071] Figure 2 is based on Figure 1 A schematic diagram of an image fusion model shown in the embodiment;

[0072] Figure 3 is a block diagram of an image fusion device according to an exemplary embodiment;

[0073] Figure 4 It is a schematic diagram of a functional block diagram of a vehicle according to an exemplary embodiment. DETAILED DESCRIPTION

[0074] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0075] Before introducing the image fusion method, device, vehicle and storage medium shown in the present disclosure, the application scenarios involved in each embodiment of the present disclosure are first introduced. In the field of autonomous driving, multiple image acquisition devices are usually set on the vehicle to obtain information around the vehicle body, and the acquisition areas of multiple image acquisition devices may overlap, resulting in the same object appearing in multiple images. At present, different models are usually used to process images captured by different image acquisition devices, but this processing method cannot accurately process objects that appear repeatedly in multiple images.

[0076] Figure 1 is a flow chart of a method for image fusion according to an exemplary embodiment. Figure 1 As shown, the method comprises the following steps:

[0077] In step S101, images captured by multiple image acquisition devices are acquired.

[0078] In step S102, image features of each image are obtained.

[0079] For example, the vehicle may first obtain multiple images captured by multiple image acquisition devices. Two or more of the multiple images may contain the same image information, that is, the acquisition areas of the image acquisition devices overlap. For the multiple images collected, the image features of each image may be extracted through a preset feature extraction model. The feature extraction model may be, for example, a VGG network model, a Resnet network model, and the like.

[0080] In step S103, the image features of each image are used as the input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model. The image fusion model is used to splice multiple image features into a target feature vector, encode the target feature vector to obtain an encoded encoding vector, and decode the encoding vector to obtain the target image.

[0081] For example, an image fusion model can be obtained in advance based on Transformer model training. After obtaining the image features corresponding to multiple images, the multiple image features can be input into the pre-trained image fusion model, and the multiple image features can be processed by the image fusion model to obtain a fused target image. In some embodiments, the image fusion model can first splice the input multiple image features into a target feature vector through a preset splicing function, where the splicing function can be, for example, a Concat function. The target feature vector can then be encoded to obtain an encoded encoding vector. In some embodiments, the format of the encoding vector is consistent with the format of the Key vector, Value vector, and Query vector in the preset Transformer model, which facilitates the image fusion model to decode the encoding vector. The encoding method of the target feature vector can be one or more of pooling, dimensionality transformation, position encoding, linear transformation, and the like.

[0082] In some embodiments, reference Figure 2, Transformer Feature Fusion can be an image fusion model, extract feature 1, extract feature 2, ..., extract feature N can be image features of multiple images, extract feature 1, extract feature 2, ..., extract feature N can be input into the concat module of Transformer Feature Fusion, and multiple image features can be spliced ​​into a target feature vector through the concat module. The target feature vector output by the concat module can be obtained by the encoding process of the first branch to obtain a first encoding vector, and by the encoding process of the second branch to obtain a second encoding vector. The encoding process of the first branch can be: the target feature vector is transformed in dimension through the Flatten module and the Linear module in sequence, and then the transformed target feature vector is positionally encoded to obtain a first position vector pos embed, and then the transformed target feature vector is added to the first position vector through the Add module to obtain the first encoding vector. The encoding process of the second branch can be: the target feature vector is pooled through the Pool module, and then the transformed target feature vector is positionally encoded to obtain a second position vector pos embed, and then the preset query vector, the transformed target feature vector and the second position vector are added through the Add module to obtain the second encoding vector. Furthermore, the first encoding vector and the second encoding vector are input into the decoding module Transformer Decoder*N for decoding, thereby obtaining a target image. Finally, the target image is output in a preset coordinate system to obtain a target feature space, and the target feature space is applied to the corresponding Head task.

[0083] In summary, the present disclosure first obtains images collected by multiple image acquisition devices, and obtains the image features of each image. The image features of each image are then used as the input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model, wherein the image fusion model is used to splice multiple image features into a target feature vector, encode the target feature vector, obtain an encoded encoding vector, and decode the encoding vector to obtain a target image. The present disclosure encodes and decodes multiple images collected by multiple image acquisition devices through an image fusion model to obtain a target image after the fusion of multiple images, which can improve the accuracy of image fusion.

[0084] In one application scenario, the image fusion model includes an encoding module and a decoding module, and the encoding module is used to:

[0085] The target feature vector is encoded according to a first encoding method to obtain a first encoding vector.

[0086] The target feature vector is encoded according to a second encoding method to obtain a second encoding vector.

[0087] The decoding module is used to:

[0088] The first coding vector and the second coding vector are decoded to obtain a target image.

[0089] For example, the encoding module can encode the target feature vector in two different ways at the same time to obtain two encoding vectors. In some embodiments, the target feature vector can be encoded in a first encoding method to obtain a first encoding vector, wherein the first encoding vector can be a Memory vector, and the Memory vector can include a Key vector and a Value vector. In other embodiments, the target feature vector can be encoded in a second encoding method to obtain a second encoding vector, wherein the second encoding vector can be a Query vector. Afterwards, the first encoding vector and the second encoding module obtained by the encoding module can be input into the decoding module, and the first encoding vector and the second encoding module are decoded by the decoding module to obtain the target image.

[0090] In another application scenario, the encoding module is used to:

[0091] Perform dimension transformation on the target feature vector to obtain a first transformed feature vector.

[0092] Position encoding is performed on the first transformed feature vector to obtain a first position vector.

[0093] The sum of the first transformed feature vector and the first position vector is used as the first encoding vector.

[0094] For example, the encoding module may first perform a dimension transformation on the target feature vector to convert the target feature vector into a first transformed feature vector in a first preset vector format. In some embodiments, the first transformed feature vector may be positionally encoded, and the position encoding may be in the form of absolute position encoding or relative position encoding, for example, position encoding may be performed using sine and cosine functions. The first transformed feature vector is added to the first position vector to embed the first position vector into the first transformed feature vector to obtain a first encoding vector, so that each element in each first transformed feature vector has a corresponding position encoding.

[0095] In another application scenario, the encoding module is used to:

[0096] The height and width of the target feature vector are flattened to obtain a first intermediate feature vector.

[0097] When the number of dimensions in the first intermediate feature vector is not equal to the preset number of dimensions, a linear transformation is performed on the first intermediate feature vector to obtain a first transformed feature vector.

[0098] For example, the target feature vector may be [B, N, C, H, W], where B is the batch size, i.e., the number of images, N is the number of image acquisition devices, C is the output dimension of the feature extraction model, H is the height of the image, i.e., the number of pixels in the vertical dimension of the image, and W is the width of the image, representing the number of pixels in the horizontal dimension of the image. The encoding module may flatten the height and width of the target feature vector by a preset flattening function, i.e., flatten the height and width of multiple images to obtain a first intermediate feature vector, where the preset flattening function may be a Flatten function, and the first intermediate feature vector may be [B, C, (N*H*W)].

[0099] In some embodiments, if the number of dimensions in the first intermediate feature vector is not equal to the preset number of dimensions, indicating that the first intermediate feature vector does not conform to the first preset vector format, then the first intermediate feature vector can be linearly transformed by a preset linear change function to obtain a first transformed feature vector that conforms to the first preset vector format, wherein the preset number of dimensions can be understood as the input dimension of the decoding module, and the preset linear change function can be a Linear function.

[0100] In another application scenario, the encoding module is used to:

[0101] When the number of dimensions in the first intermediate feature vector is equal to the preset number of dimensions, the first intermediate feature vector is used as the first transformed feature vector.

[0102] For example, if the number of dimensions in the first intermediate feature vector is equal to the preset number of dimensions, it means that the first intermediate feature vector conforms to the first preset vector format, and the first intermediate feature vector can be directly used as the first transformed feature vector.

[0103] In another application scenario, the encoding module is used to:

[0104] The target feature vector is transformed in dimension to obtain a second transformed feature vector.

[0105] Position encoding is performed on the second transformed feature vector to obtain a second position vector.

[0106] The sum of the second transformed feature vector, the second position vector and the preset query vector is used as the second encoding vector.

[0107] For example, the encoding module may first perform a dimension transformation on the target feature vector, and convert the target feature vector into a second transformed feature vector in a second preset vector format. In some embodiments, the second transformed feature vector may be positionally encoded, and the position encoding may be in the form of absolute position encoding or relative position encoding, for example, position encoding may be performed using sine and cosine functions. The second transformed feature vector, the second position vector, and the preset query vector are added, and the second position vector is embedded in the second transformed feature vector to obtain a second encoding vector, so that each element in each second transformed feature vector has a corresponding position encoding.

[0108] In another application scenario, the encoding module is used to:

[0109] The target feature vector is pooled to obtain a second intermediate feature vector.

[0110] When the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is not equal to the number of dimensions in the query vector, a linear transformation is performed on the second intermediate feature vector to obtain a second transformed feature vector.

[0111] For example, the encoding module can perform pooling processing on the target feature vector to change the dimension of the target feature vector to obtain a second intermediate feature vector. The pooling processing can be average pooling (Avg-pooling) or maximum pooling (Max-pooling) and other pooling methods are not specifically limited in the present disclosure. In some embodiments, if the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is not equal to the number of dimensions in the query vector, indicating that the second intermediate feature vector does not conform to the second preset vector format, then the second intermediate feature vector can be linearly transformed by a preset linear change function to obtain a second transformed feature vector that conforms to the second preset vector format, wherein the preset linear change function can be a Linear function.

[0112] In another application scenario, the encoding module is used to:

[0113] When the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is equal to the number of dimensions in the query vector, the second intermediate feature vector is used as the second transformed feature vector.

[0114] For example, if the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is equal to the number of dimensions in the query vector, indicating that the second intermediate feature vector conforms to the second preset vector format, then the second intermediate feature vector can be directly used as the second transformed feature vector.

[0115] In summary, the present disclosure first obtains images collected by multiple image acquisition devices, and obtains the image features of each image. The image features of each image are then used as the input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model, wherein the image fusion model is used to splice multiple image features into a target feature vector, encode the target feature vector, obtain an encoded encoding vector, and decode the encoding vector to obtain a target image. The present disclosure encodes and decodes multiple images collected by multiple image acquisition devices through an image fusion model to obtain a target image after the fusion of multiple images, which can improve the accuracy of image fusion.

[0116] Figure 3 is a block diagram of an image fusion device according to an exemplary embodiment. Figure 3 As shown, the device 200 includes:

[0117] The first acquisition module 201 is configured to acquire images acquired by multiple image acquisition devices.

[0118] The second acquisition module 202 is configured to acquire image features of each image.

[0119] The fusion module 203 is used to use the image features of each image as the input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model. The image fusion model is used to splice multiple image features into a target feature vector, encode the target feature vector to obtain an encoded encoding vector, and decode the encoding vector to obtain the target image.

[0120] In one application scenario, the image fusion model includes an encoding module and a decoding module. The encoding module is used to:

[0121] The target feature vector is encoded according to a first encoding method to obtain a first encoding vector.

[0122] The target feature vector is encoded according to a second encoding method to obtain a second encoding vector.

[0123] The decoding module is used to:

[0124] The first coding vector and the second coding vector are decoded to obtain a target image.

[0125] In another application scenario, the encoding module is used to:

[0126] Perform dimension transformation on the target feature vector to obtain a first transformed feature vector.

[0127] Position encoding is performed on the first transformed feature vector to obtain a first position vector.

[0128] The sum of the first transformed feature vector and the first position vector is used as the first encoding vector.

[0129] In another application scenario, the encoding module is used to:

[0130] The height and width of the target feature vector are flattened to obtain a first intermediate feature vector.

[0131] When the number of dimensions in the first intermediate feature vector is not equal to the preset number of dimensions, a linear transformation is performed on the first intermediate feature vector to obtain a first transformed feature vector.

[0132] In another application scenario, the encoding module is used to:

[0133] When the number of dimensions in the first intermediate feature vector is equal to the preset number of dimensions, the first intermediate feature vector is used as the first transformed feature vector.

[0134] In another application scenario, the encoding module is used to:

[0135] The target feature vector is transformed in dimension to obtain a second transformed feature vector.

[0136] Position encoding is performed on the second transformed feature vector to obtain a second position vector.

[0137] The sum of the second transformed feature vector, the second position vector and the preset query vector is used as the second encoding vector.

[0138] In another application scenario, the encoding module is used to:

[0139] The target feature vector is pooled to obtain a second intermediate feature vector.

[0140] When the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is not equal to the number of dimensions in the query vector, a linear transformation is performed on the second intermediate feature vector to obtain a second transformed feature vector.

[0141] In another application scenario, the encoding module is used to:

[0142] When the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is equal to the number of dimensions in the query vector, the second intermediate feature vector is used as the second transformed feature vector.

[0143] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0144] In summary, the present disclosure first obtains images collected by multiple image acquisition devices, and obtains the image features of each image. The image features of each image are then used as the input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model, wherein the image fusion model is used to splice multiple image features into a target feature vector, encode the target feature vector, obtain an encoded encoding vector, and decode the encoding vector to obtain a target image. The present disclosure encodes and decodes multiple images collected by multiple image acquisition devices through an image fusion model to obtain a target image after the fusion of multiple images, which can improve the accuracy of image fusion.

[0145] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, and when the program instructions are executed by a processor, the steps of the image fusion method provided by the present disclosure are implemented.

[0146] See also Figure 4 , Figure 4 3 is a functional block diagram of a vehicle 300 shown in an exemplary embodiment. The vehicle 300 can be configured in a fully or partially autonomous driving mode. For example, the vehicle 300 can obtain environmental information about its surroundings through the perception system 320, and obtain an autonomous driving strategy based on the analysis of the surrounding environmental information to achieve fully autonomous driving, or present the analysis results to the user to achieve partially autonomous driving.

[0147] The vehicle 300 may include various subsystems, such as an infotainment system 310, a perception system 320, a decision control system 330, a drive system 340, and a computing platform 350. Optionally, the vehicle 300 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of the vehicle 300 may be interconnected by wire or wirelessly.

[0148] In some embodiments, infotainment system 310 may include communication system 311 , entertainment system 312 , and navigation system 313 .

[0149] The communication system 311 may include a wireless communication system that can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system may use 3G cellular communications, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communications, such as LTE. Or 5G cellular communications. The wireless communication system may communicate with a wireless local area network (WLAN) using WiFi. In some embodiments, the wireless communication system may communicate directly with the device using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, for example, the wireless communication system may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0150] The entertainment system 312 may include a display device, a microphone and speakers. Users can listen to the radio and play music in the car based on the entertainment system; or connect the mobile phone to the vehicle and project the mobile phone's screen on the display device. The display device may be touch-screen, and users can operate by touching the screen.

[0151] In some cases, the user's voice signal can be obtained through a microphone, and based on the analysis of the user's voice signal, the user can control certain aspects of the vehicle 300, such as adjusting the temperature inside the vehicle, etc. In other cases, music can be played to the user through a speaker.

[0152] The navigation system 313 may include a map service provided by a map provider, thereby providing navigation of the driving route for the vehicle 300. The navigation system 313 may be used in conjunction with the vehicle's global positioning system 321 and inertial measurement unit 322. The map service provided by the map provider may be a two-dimensional map or a high-precision map.

[0153] The perception system 320 may include several sensors that sense information about the environment around the vehicle 300. For example, the perception system 320 may include a global positioning system 321 (the global positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU) 322, a laser radar 323, a millimeter wave radar 324, an ultrasonic radar 325, and a camera 326. The perception system 320 may also include sensors of the internal systems of the monitored vehicle 300 (for example, an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). This detection and recognition is a key function for the safe operation of the vehicle 300.

[0154] The global positioning system 321 is used to estimate the geographic location of the vehicle 300 .

[0155] The inertial measurement unit 322 is used to sense the change of the posture of the vehicle 300 based on inertial acceleration. In some embodiments, the inertial measurement unit 322 may be a combination of an accelerometer and a gyroscope.

[0156] Lidar 323 utilizes laser light to sense objects in the environment in which vehicle 300 is located. In some embodiments, Lidar 323 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components.

[0157] The millimeter wave radar 324 uses radio signals to sense objects in the surrounding environment of the vehicle 300. In some embodiments, in addition to sensing objects, the millimeter wave radar 324 can also be used to sense the speed and / or heading of the object.

[0158] The ultrasonic radar 325 may sense objects around the vehicle 300 using ultrasonic signals.

[0159] The camera device 326 is used to capture image information of the surrounding environment of the vehicle 300. The camera device 326 may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, etc. The image information acquired by the camera device 326 may include a static image or video stream information.

[0160] The decision control system 330 includes a computing system 331 that analyzes and makes decisions based on the information obtained by the perception system 320. The decision control system 330 also includes a vehicle controller 332 that controls the power system of the vehicle 300, and a steering system 333, a throttle 334 and a braking system 335 for controlling the vehicle 300.

[0161] The computing system 331 can be operated to process and analyze various information obtained by the perception system 320 in order to identify targets, objects and / or features in the environment surrounding the vehicle 300. Targets may include pedestrians or animals, and objects and / or features may include traffic signals, road boundaries and obstacles. The computing system 331 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking and other technologies. In some embodiments, the computing system 331 can be used to map the environment, track objects, estimate the speed of objects, and so on. The computing system 331 can analyze the various information obtained and derive a control strategy for the vehicle.

[0162] The vehicle controller 332 can be used to coordinate and control the vehicle's power battery and engine 341 to improve the power performance of the vehicle 300.

[0163] The steering system 333 is operable to adjust the forward direction of the vehicle 300. For example, in one embodiment, it may be a steering wheel system.

[0164] The throttle 334 is used to control the operating speed of the engine 341 and thus the speed of the vehicle 300 .

[0165] Braking system 335 is used to control the deceleration of vehicle 300. Braking system 335 can use friction to slow down wheels 344. In some embodiments, braking system 335 can convert the kinetic energy of wheels 344 into electric current. Braking system 335 can also take other forms to slow down the rotation speed of wheels 344 to control the speed of vehicle 300.

[0166] The drive system 340 may include components that provide powered movement for the vehicle 300. In one embodiment, the drive system 340 may include an engine 341, an energy source 342, a transmission system 343, and wheels 344. The engine 341 may be an internal combustion engine, an electric motor, an air compression engine, or a combination of other types of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine. The engine 341 converts the energy source 342 into mechanical energy.

[0167] Examples of energy source 342 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 342 may also provide energy to other systems of vehicle 300.

[0168] The transmission system 343 can transmit the mechanical power from the engine 341 to the wheels 344. The transmission system 343 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission system 343 may also include other devices, such as a clutch. Among them, the drive shaft may include one or more shafts that can be coupled to one or more wheels 344.

[0169] Some or all functions of the vehicle 300 are controlled by a computing platform 350. The computing platform 350 may include at least one processor 351 that may execute instructions 353 stored in a non-transitory computer-readable medium such as a memory 352. In some embodiments, the computing platform 350 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 300 in a distributed manner.

[0170] The processor 351 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor 351 may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof. Figure 4 Functionally illustrated processor, memory and other elements of the computer in the same block, but those of ordinary skill in the art will appreciate that the processor, computer or memory may actually include multiple processors, computers or memories that may or may not be stored in the same physical housing. For example, the memory may be a hard drive or other storage media that is located in a housing different from the computer. Therefore, references to processors or computers will be understood to include references to a collection of processors or computers or memories that may or may not operate in parallel. Different from using a single processor to perform the steps described herein, some components such as steering components and deceleration components may each have their own processors that only perform calculations related to the functions specific to the components.

[0171] In the embodiment of the present disclosure, the processor 351 may execute the above-mentioned image fusion method.

[0172] In various aspects described herein, processor 351 may be located remote from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle and others are performed by a remote processor, including taking the necessary steps to perform a single maneuver.

[0173] In some embodiments, the memory 352 may include instructions 353 (e.g., program logic) that may be executed by the processor 351 to perform various functions of the vehicle 300. The memory 352 may also include additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the infotainment system 310, the perception system 320, the decision control system 330, and the drive system 340.

[0174] In addition to instructions 353, memory 352 may also store data such as road maps, route information, the vehicle's location, direction, speed, and other such vehicle data, as well as other information. Such information may be used by vehicle 300 and computing platform 350 during operation of vehicle 300 in autonomous, semi-autonomous, and / or manual modes.

[0175] The computing platform 350 may control functions of the vehicle 300 based on input received from various subsystems, such as the drive system 340, the perception system 320, and the decision control system 330. For example, the computing platform 350 may utilize input from the decision control system 330 in order to control the steering system 333 to avoid an obstacle detected by the perception system 320. In some embodiments, the computing platform 350 may be operable to provide control over many aspects of the vehicle 300 and its subsystems.

[0176] Alternatively, one or more of the above-mentioned components may be installed or associated separately from the vehicle 300. For example, the memory 352 may exist partially or completely separately from the vehicle 300. The above-mentioned components may be communicatively coupled together in a wired and / or wireless manner.

[0177] Optionally, the above components are only examples. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 4 It should not be understood as limiting the embodiments of the present disclosure.

[0178] An autonomous vehicle traveling on a road, such as vehicle 300 above, can identify objects in its surroundings to determine adjustments to the current speed. Objects can be other vehicles, traffic control devices, or other types of objects. In some examples, each identified object can be considered independently, and based on the object's respective characteristics, such as its current speed, acceleration, spacing from the vehicle, etc., can be used to determine the speed to be adjusted by the autonomous vehicle.

[0179] Optionally, the vehicle 300 or a sensing and computing device associated with the vehicle 300 (e.g., computing system 331, computing platform 350) can predict the behavior of the identified object based on the characteristics of the identified object and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so all the identified objects can also be considered together to predict the behavior of a single identified object. The vehicle 300 can adjust its speed based on the predicted behavior of the identified object. In other words, the autonomous vehicle can determine what stable state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the object. In this process, other factors can also be considered to determine the speed of the vehicle 300, such as the lateral position of the vehicle 300 in the road it is traveling on, the curvature of the road, the proximity of static and dynamic objects, etc.

[0180] In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device may also provide instructions to modify the steering angle of vehicle 300 so that the autonomous vehicle follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the autonomous vehicle (e.g., vehicles in adjacent lanes on the road).

[0181] The vehicle 300 may be any type of travel vehicle, such as a car, a truck, a motorcycle, a bus, a ship, an airplane, a helicopter, an amusement vehicle, a train, etc., and the present disclosure does not specifically limit the above-mentioned embodiments.

[0182] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-mentioned image fusion method when executed by the programmable device.

[0183] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0184] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for image fusion, characterized in that: The method comprises: Acquire images acquired by a plurality of image acquisition devices, wherein the plurality of image acquisition devices are used to acquire image information around the vehicle, and acquisition areas of the plurality of image acquisition devices partially overlap and partially do not overlap; Obtain image features for each image; Using the image features of each image as the input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model, wherein the image fusion model is used to splice a plurality of the image features into a target feature vector, encode the target feature vector to obtain an encoded encoding vector, and decode the encoding vector to obtain the target image; The image fusion model includes an encoding module and a decoding module, and the encoding module is used to: Encoding the target feature vector according to a first encoding method to obtain a first encoding vector; Encoding the target feature vector according to a second encoding method to obtain a second encoding vector; The decoding module is used for: Decoding the first coding vector and the second coding vector to obtain the target image; The encoding of the target feature vector according to the first encoding mode to obtain the first encoding vector comprises: Performing a dimension transformation on the target feature vector to obtain a first transformed feature vector; Performing position encoding on the first transformed feature vector to obtain a first position vector; Taking the sum of the first transformed feature vector and the first position vector as the first encoding vector; The encoding of the target feature vector according to the second encoding mode to obtain the second encoding vector comprises: Performing a dimension transformation on the target feature vector to obtain a second transformed feature vector; Performing position encoding on the second transformed feature vector to obtain a second position vector; The sum of the second transformed feature vector, the second position vector and a preset query vector is used as the second encoding vector.

2. The method according to claim 1, characterized in that The performing dimension transformation on the target feature vector to obtain a first transformed feature vector comprises: Flattening the height and width of the target feature vector to obtain a first intermediate feature vector; When the number of dimensions in the first intermediate feature vector is not equal to the preset number of dimensions, a linear transformation is performed on the first intermediate feature vector to obtain the first transformed feature vector.

3. The method according to claim 2, characterized in that The performing dimension transformation on the target feature vector to obtain a first transformed feature vector further comprises: When the number of dimensions in the first intermediate feature vector is equal to the preset number of dimensions, the first intermediate feature vector is used as the first transformed feature vector.

4. The method according to claim 1, characterized in that: The performing dimension transformation on the target feature vector to obtain a second transformed feature vector comprises: Performing pooling processing on the target feature vector to obtain a second intermediate feature vector; When the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is not equal to the number of dimensions in the query vector, a linear transformation is performed on the second intermediate feature vector to obtain the second transformed feature vector.

5. The method according to claim 4, characterized in that The performing dimension transformation on the target feature vector to obtain a second transformed feature vector further comprises: When the product of the number of image acquisition devices in the second intermediate feature vector and the number of dimensions in the second intermediate feature vector is equal to the number of dimensions in the query vector, the second intermediate feature vector is used as the second transformed feature vector.

6. An image fusion device, characterized in that: The device comprises: A first acquisition module is configured to acquire images acquired by a plurality of image acquisition devices, wherein the plurality of image acquisition devices are used to acquire image information around the vehicle, and acquisition areas of the plurality of image acquisition devices partially overlap and partially do not overlap; A second acquisition module is configured to acquire image features of each image; A fusion module, used to use the image features of each image as input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model, wherein the image fusion model is used to splice a plurality of the image features into a target feature vector, encode the target feature vector to obtain an encoded encoding vector, and decode the encoding vector to obtain the target image; The image fusion model includes an encoding module and a decoding module, and the encoding module is used to: Encoding the target feature vector according to a first encoding method to obtain a first encoding vector; Encoding the target feature vector according to a second encoding method to obtain a second encoding vector; The decoding module is used for: Decoding the first coding vector and the second coding vector to obtain the target image; The encoding module is used for: Performing a dimension transformation on the target feature vector to obtain a first transformed feature vector; Performing position encoding on the first transformed feature vector to obtain a first position vector; Taking the sum of the first transformed feature vector and the first position vector as the first encoding vector; Performing a dimension transformation on the target feature vector to obtain a second transformed feature vector; Performing position encoding on the second transformed feature vector to obtain a second position vector; The sum of the second transformed feature vector, the second position vector and a preset query vector is used as the second encoding vector.

7. A vehicle, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Acquire images acquired by a plurality of image acquisition devices, wherein the plurality of image acquisition devices are used to acquire image information around the vehicle, and acquisition areas of the plurality of image acquisition devices partially overlap and partially do not overlap; Obtain image features for each image; Using the image features of each image as the input of a pre-trained image fusion model to obtain a fused target image output by the image fusion model, wherein the image fusion model is used to splice a plurality of the image features into a target feature vector, encode the target feature vector to obtain an encoded encoding vector, and decode the encoding vector to obtain the target image; The image fusion model includes an encoding module and a decoding module, and the encoding module is used to: Encoding the target feature vector according to a first encoding method to obtain a first encoding vector; Encoding the target feature vector according to a second encoding method to obtain a second encoding vector; The decoding module is used for: Decoding the first coding vector and the second coding vector to obtain the target image; The encoding of the target feature vector according to the first encoding mode to obtain the first encoding vector comprises: Performing a dimension transformation on the target feature vector to obtain a first transformed feature vector; Performing position encoding on the first transformed feature vector to obtain a first position vector; Taking the sum of the first transformed feature vector and the first position vector as the first encoding vector; The encoding of the target feature vector according to the second encoding mode to obtain the second encoding vector comprises: Performing a dimension transformation on the target feature vector to obtain a second transformed feature vector; Performing position encoding on the second transformed feature vector to obtain a second position vector; The sum of the second transformed feature vector, the second position vector and a preset query vector is used as the second encoding vector.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

Images