Data processing method and device and storage medium

Abstract

The invention relates to a data processing method and device and a storage medium. The data processing method disclosed by the invention comprises the following steps: receiving sensor data about an environment collected by a sensor; generating a first feature map according to the sensor data; a second feature map used for describing the environment is updated according to the first feature map, each feature in the first feature map and the second feature map has position information used for indicating the position, corresponding to the feature, in the environment, and the second feature map used for describing the environment is updated according to the first feature map and used for indicating the position, corresponding to the feature, in the environment. Comprising the following steps: for each feature in at least one feature in a second feature map, determining a plurality of features in a first feature map according to the position information of the feature in the second feature map and the position information of the feature in the first feature map; and updating the features in the second feature map according to the determined multiple features in the first feature map.

Description

Technical Field

[0001] The present disclosure relates to a data processing method, device and storage medium. Background Art

[0002] In the field of autonomous driving perception, it is usually necessary to fuse multimodal data collected from different sensors so that perception can be better based on fused features. It is hoped that a unified representation space can be used to achieve cross-modal data fusion. How to convert sensor data into a specific representation space is a challenge. Summary of the invention

[0003] The present disclosure provides a data processing method, device and storage medium, which can convert sensor data into a specific representation space.

[0004] The present disclosure provides, in one aspect, a data processing method, comprising:

[0005] receiving sensor data about the environment collected by the sensor;

[0006] generating a first feature map according to the sensor data;

[0007] updating a second feature map for describing the environment according to the first feature map,

[0008] Each feature in the first feature map and the second feature map has position information, which is used to indicate the position in the environment corresponding to the feature.

[0009] Wherein, updating the second feature map used to describe the environment according to the first feature map includes:

[0010] For each feature in at least one feature in the second feature map:

[0011] Determine a plurality of features in the first feature map according to the position information of the feature in the second feature map and the position information of the feature in the first feature map;

[0012] The feature in the second feature map is updated according to the determined multiple features in the first feature map.

[0013] In another aspect, the present disclosure provides an apparatus for data processing, comprising:

[0014] Processor; and

[0015] A memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the data processing method according to the present disclosure.

[0016] In yet another aspect, the present disclosure provides a computer-readable storage medium storing a program, wherein the program includes instructions, which, when executed by one or more processors of a computing device, cause the computing device to execute the data processing method according to the present disclosure.

[0017] In the field of perception of autonomous driving, it is usually necessary to fuse multimodal data collected from sensors with different perspectives so that perception can be better based on fusion features. It is hoped that a unified representation space can be used to achieve cross-modal data fusion. For example, the sensor data is projected into the Bird's Eye View (BEV) space, so that sensor data of different perspectives and different modalities are integrated in a unified representation space. However, many sensor data are difficult to project into a specific representation space. For example, the original RADAR data without complete distance and angle solution cannot explicitly express all spatial information, and traditional technology cannot use BEV space to fuse the original RADAR data. The data processing method disclosed in the present invention includes receiving sensor data about the environment collected by the sensor; generating a first feature map according to the sensor data; and updating a second feature map for describing the environment according to the first feature map. According to the data processing method disclosed in the present invention, the target task can be performed according to the second feature map. The data processing method disclosed in the present invention can convert various sensor data (such as original RADAR data) into a specific representation space (i.e., a unified representation space, such as BEV space), so that the feature fusion of various sensor data can be better achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 is a schematic diagram of a vehicle located in an environment according to an example embodiment of the present disclosure;

[0020] Figure 2 is a schematic diagram of a computing device according to an example embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram of a data processing apparatus according to an exemplary embodiment of the present disclosure;

[0022] Figure 4 is a schematic diagram of an update operation according to an exemplary embodiment of the present disclosure;

[0023] Figure 5 is a schematic diagram of an update operation according to another exemplary embodiment of the present disclosure;

[0024] Fig. 6A is a schematic diagram of a data processing system according to an example embodiment of the present disclosure;

[0025] Figure 6B is a schematic diagram of a data processing system according to another example embodiment of the present disclosure;

[0026] Figure 7 is a structural diagram of a task execution module provided according to an exemplary embodiment of the present disclosure;

[0027] Figure 8 is a flow chart of a data processing method according to an exemplary embodiment of the present disclosure;

[0028] Fig. 9 is a flowchart of a data processing method according to another exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] In this disclosure, the term "plurality" means two or more, unless otherwise specified. In this disclosure, the term "and / or" describes the association relationship of associated objects, covering any one of the listed objects and all possible combinations. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0031] In the present disclosure, unless otherwise specified, the terms "first", "second", etc. are used to distinguish similar objects without intending to limit their positional relationship, temporal relationship, or importance relationship. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in ways other than those illustrated or described herein.

[0032] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, system, product, or apparatus.

[0033] Figure 1 1 is a schematic diagram of a vehicle 120 located in an environment 100 according to an example embodiment of the present disclosure. The vehicle 120 may be a car, a truck, a motorcycle, a bus, a recreational vehicle, an amusement park vehicle, a tram, a golf cart, a train, a trolley bus, or other vehicle. The vehicle 120 may be operated in an autonomous driving mode, in whole or in part. The vehicle 120 may control itself in the autonomous driving mode, for example, the vehicle 120 may determine a current state of the vehicle 120 and a current state of the environment in which the vehicle 120 is located, determine a predicted behavior of at least one other vehicle in the environment, determine a confidence level corresponding to the likelihood that the at least one other vehicle will perform the predicted behavior, and control the vehicle 120 itself based on the determined information. While in the autonomous driving mode, the vehicle 120 may operate without human interaction.

[0034] The vehicle 120 may include various sensors for sensing the environment 100 around the vehicle and the conditions of the vehicle itself, such as an inertial measurement unit (IMU), a global navigation satellite system (GNSS) transceiver (e.g., a global positioning system (GPS) transceiver), a radio detection and ranging (RADAR) sensor, a laser detection and ranging (LiDAR) sensor, an acoustic sensor, an ultrasonic sensor, and one or more of a camera. These sensors may be disposed at any position of the vehicle 120 as required.

[0035] One or more of these sensors can be driven individually or collectively to update the pose (e.g., position and orientation) of one or more sensors. Some sensors can have overlapping fields of view, so that at or about the same time, these sensors can collect data about the same scene content. As an example, Figure 1 , a camera 121 and a RADAR 122 are shown with their fields of view being located in front of a vehicle 120 , and the camera 121 and the RADAR 122 have overlapping fields of view.

[0036] The vehicle 120 may also include a computing device 130. The computing device can control some or all functions of the vehicle 120. The computing device may include, for example, an autonomous driving control unit, which may be composed of one or more algorithm modules, including various machine learning models (e.g., neural networks) that can perform real-time or onboard processing (e.g., object detection) on data collected by sensors on the vehicle 120 to identify, evaluate, and avoid or overcome potential obstacles in the environment 100 where the vehicle is located. In some embodiments, the autonomous driving control unit is used to combine data from sensors to determine the driving path or trajectory of the vehicle 120.

[0037] Figure 22 is a schematic diagram of a computing device 200 according to an exemplary embodiment of the present application. The computing device 200 may be, for example, Figure 1 An example of a computing device 130 is shown. The computing device 200 can be a server, a personal computer (PC), a laptop computer, a tablet computer, a personal digital assistant (PDA), an electronic control unit (ECU), a cellular phone, a smart phone, a set-top box (STB), etc. The example computing device 200 may include a processor 202 (e.g., a system on a chip (SoC), a general processing core, a graphics core, and optional other processing logic) and a memory 204 that can communicate with each other via a bus 206 or other data transmission system. The computing device 200 may also include various input / output (I / O) devices or interfaces 210 (such as a touch screen display, an audio jack, a voice interface) and an optional network interface 212. The network interface 212 can use 3G cellular communications (e.g., CDMA, EVDO, GSM / GPRS) or 4G cellular communications (e.g., WiMAX or LTE), and can also use 5G cellular communications. Optionally, the network interface 212 can communicate with a wireless local area network (WLAN) (e.g., using ). In an example embodiment, the network interface 212 may include or support substantially any wired and / or wireless communication and data processing mechanism by which information / data may be communicated between the computing device 200 and another computing device or system via the network 214.

[0038] The memory 204 is an example of a computer-readable storage medium on which is stored one or more instruction sets, software, firmware, or other processing logic (e.g., logic 208) that implement any one or more of the methodologies or functions described and / or claimed herein. The logic 208, or a portion thereof, may also reside, in whole or in part, within the processor 202 during execution by the computing device 200.

[0039] The term computer-readable storage medium may be understood to include a single non-transitory medium or multiple non-transitory media (e.g., centralized or distributed databases and / or associated caches and computing systems) that store one or more instruction sets. The term computer-readable storage medium may also be understood to include any non-transitory medium that can store, encode, or carry an instruction set for execution by a machine and cause the machine to perform any one or more of the methods of the various embodiments or that can store, encode, or carry a data structure utilized by or associated with such an instruction set. The term computer-readable storage medium may therefore be understood to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0040] In the scenario of autonomous driving, the use of point clouds collected by LiDAR and images collected by cameras for three-dimensional (3D) target positioning and recognition is a mainstream paradigm. By utilizing the rich semantic information provided by images and the precise spatial information provided by point clouds, good detection results can be achieved. However, this method also has obvious defects. When encountering bad weather, such as fog, rain and snow, the perception capabilities of LiDAR and cameras will be seriously disturbed, resulting in inaccurate position measurement and target recognition, making it difficult to ensure the safety of autonomous driving vehicles. In addition, the speed information of traffic participants is very important for the autonomous driving control unit to make judgments on lane changes, acceleration and deceleration, etc. Neither LiDAR nor cameras can directly measure the moving speed of the target, which will cause certain defects in the timeliness and accuracy of the decision-making of the autonomous driving control unit, thereby affecting driving safety and system performance.

[0041] The above problems associated with LiDAR and cameras can be solved by RADAR-based 3D target detection. Given that point cloud-based 3D target detection technology is relatively mature, traditional RADAR-based 3D target detection methods directly detect targets based on point cloud data provided by RADAR and in combination with other modalities, thereby taking advantage of RADAR's robustness to bad weather and its ability to measure the radial velocity of targets. However, the acquisition of RADAR point cloud data requires sidelobe suppression and a constant false alarm rate (CFAR) detector to remove noise and false alarms, which will lose a lot of information about the surrounding environment. The resulting RADAR point cloud is also extremely sparse, which makes it very difficult to locate and identify targets, especially long-distance targets, and is not as effective as the target detection algorithm based on LiDAR point cloud.

[0042] In addition, in the field of perception of autonomous driving, it is usually necessary to fuse multimodal data collected from sensors with different perspectives so that perception can be better based on fusion features. It is hoped that a unified representation space can be used to achieve cross-modal data fusion. For example, sensor data can be projected into the BEV space (the space from a bird's-eye view is called the BEV space) to integrate data from different perspectives and different modalities in a unified representation space. However, many sensor data face difficulties in projecting into a specific representation space. For example, raw RADAR data that has not undergone complete distance and angle resolution cannot explicitly express all spatial information, and traditional methods cannot project raw RADAR data into a specific representation space (such as the BEV space).

[0043] Figure 3is a schematic diagram of a data processing apparatus 300 according to an exemplary embodiment of the present application. An example of the data processing apparatus 300 is a computer program implemented on one or more computing devices at one or more locations. In one example, the data processing apparatus 300 is Figure 1 The computing device 130 shown or Figure 2 The computer program implemented on the computing device 200 shown in FIG. 3 can generate a feature map of a specific representation space (eg, BEV space) based on the sensor data received from the sensor, so that the sensor data of different perspectives can be projected into a unified representation space.

[0044] like Figure 3 As shown, the data processing device 300 may include a feature extraction module 320 and a feature update module 340. The data processing device 300 may receive sensor data 315 as input from one or more sensors. For example, the data processing device 300 may receive sensor data 315 from one or more sensors. Figure 1 Sensors on the vehicle 120 are shown receiving as input data collected by the sensors about the environment surrounding the vehicle 120. In one example, the sensor data 315 may include data from Figure 1 The image obtained by the camera 121 shown in FIG. 3 may include any suitable type of image, for example, a two-dimensional image with multiple channels (e.g., an RGB image). In another example, the sensor data 315 may include an image obtained from the camera 121. Figure 1 The RADAR 122 is shown acquiring RADAR data (eg, raw RADAR data).

[0045] like Figure 3 As shown, the data processing device 300 may also receive one or more feature maps 335 as input. Figure 1 The environment 100 shown in FIG. 1 is divided into a grid, and each feature in the feature map 335 may correspond to one or more grid cells of the grid. Each feature in the feature map 335 has position information, that is, information about the grid cell corresponding to the feature in the environment (e.g. Figure 1 Information about the location in the environment 100 shown.

[0046] In some embodiments, the feature map 335 is a BEV feature map, and each feature in the BEV feature map may correspond to one or more grid cells of the BEV grid, so that the BEV feature map may have different resolutions. Figure 1 The field of view of the sensor (e.g., RADAR 122) of the vehicle 120 shown is divided into a grid within a predetermined range of the sensor, resulting in a BEV grid. The grid cells of the BEV grid can be arranged in a two-dimensional pattern. Each grid cell can be regarded as an infinitely high pillar.

[0047] In some embodiments, Figure 1 The fixed points in the environment 100 are shown as poles, or Figure 1 The vehicle 120 or the sensor on the vehicle (e.g., RADAR 122) in the environment 100 shown is a pole, and a ray parallel to the ground is drawn from the pole as a polar axis, and a polar coordinate system is established on the plane where the ray is located (e.g., a plane parallel to the ground). In the present disclosure, the polar coordinate system established with RADAR as the pole is also referred to as the RADAR polar coordinate system. The position of the grid cell corresponding to the feature in the BEV feature map in the environment 100 is represented by the coordinates (r bev ,φ bev ). The features in the BEV feature map are represented by f i (r bev ,φ bev ) is expressed as follows. bev represents the polar diameter, that is, the distance between the grid unit and the pole; φ bev Represents the polar angle, which is the angle between the line segment from the pole to the grid cell and the polar axis.

[0048] In some embodiments, feature map 335 may include one or more sets of feature maps. Each set of feature maps may include one or more feature maps. Each feature map may include multiple features. In one example, each set of feature maps may include multiple feature maps with different spatial scales (or different resolutions), thereby forming a multi-scale feature map, each feature map may have its own spatial resolution.

[0049] In some embodiments, each feature map 335 is an h i × i ×c i Tensor (h i 、w i are all positive integers greater than 1, c i is a positive integer greater than or equal to 1), including the two-dimensional arrangement of h i × i c i dimensional features, each feature is a vector or other ordered set of values. That is, each feature graph 335 includes h i Line i The features of the column, each feature is a c i dimensional features (each dimension corresponds to a channel). i The value of is equal to the number of channels of the feature map 335. Different feature maps 335 may have different spatial resolutions, i.e., different h i and w i values, and / or, different numbers of channels, i.e. different c ivalue.

[0050] In some embodiments, at least one of the feature maps 335 has the same resolution as at least one of the feature maps 325 (or at least one of the feature maps in each set of feature maps 325). In some embodiments, at least one of the feature maps 335 has the same dimensionality as at least one of the feature maps 325 (or at least one of the feature maps in each set of feature maps 325). In some embodiments, at least one of the feature maps 335 has the same resolution and the same dimensionality as at least one of the feature maps 325 (or at least one of the feature maps in each set of feature maps 325).

[0051] In some embodiments, the feature map 335 is an output of another data processing device having the same structure and function as the data processing device 300. In this case, an example of the feature map 335 is Fig. 6A and 6B In some embodiments, the feature map 335 is an initialization module (e.g., Fig. 6A and 6B The output of the initialization module 620 shown in FIG. 3 is, at this time, an example of the feature map 335 is Fig. 6A and 6B Feature diagram 625 is shown.

[0052] The feature extraction module 320 can perform feature extraction on the sensor data 315 to generate one or more feature maps 325 as output. The feature extraction module 320 may include an encoder, which performs feature extraction on the sensor data 315 to generate feature maps. The encoder may include a backbone network 321. An example of a backbone network 321 is a convolutional neural network, such as a ResNet network. In an example, in addition to the backbone network 321, the encoder may also include a feature pyramid network (FPN) 322.

[0053] Each feature in the feature map 325 generated by the feature extraction module 320 has information about the location of the element of the sensor data corresponding to the feature (e.g., a pixel of an image or raw RADAR data). It should be noted that in the present disclosure, the feature map generated by performing feature extraction on an image is also referred to as an image feature map, the feature map generated by performing feature extraction on RADAR data is also referred to as a RADAR feature map, and the feature map generated by performing feature extraction on raw RADAR data is also referred to as a raw RADAR feature map. The features in the image feature map have information about the location of the pixel corresponding to the feature in the image, and the features in the RADAR feature map have information about the location of the pixel corresponding to the feature in the environment (e.g., Figure 1Information about the location in the environment 100 shown.

[0054] In some embodiments, the sensor data 315 received by the feature extraction module 320 is two-dimensional data, for example, the sensor data 315 is a two-dimensional image or raw RADAR data. The features of each feature map 325 generated by the feature extraction module 320 are arranged in a two-dimensional pattern. Each feature map 325 generated by the feature extraction module 320 can have a respective number of channels. The features in the feature map 325 can be determined by f j (x, y) represents, (x, y) represents the position of the element corresponding to the feature. Each feature map 325 generated by the feature extraction module 320 can be an h j × j ×c j Tensor (h j 、w j are all positive integers greater than 1, c j is a positive integer greater than or equal to 1), including the two-dimensional arrangement of h j × j c j dimensional features, each feature is a vector or other ordered set of values. That is, each feature map 325 includes h j Line j The features of the column, each feature is a c j dimensional features (each dimension corresponds to a channel). j The value of h is equal to the number of channels of the feature map 325. Different feature maps generated by the feature extraction module 320 may have different spatial resolutions, i.e., different h j and w j values, and / or, different numbers of channels, i.e. different c j value.

[0055] In some embodiments, the sensor data 315 may include one or more frames of sensor data (e.g., one or more images, or one or more frames of raw RADAR data). The feature map 325 may include one or more sets of feature maps. The feature extraction module 320 may generate a set of feature maps for each frame of sensor data. In this way, each set of feature maps generated by the feature extraction module 320 may correspond to a frame of sensor data. Each set of feature maps may include one or more feature maps. Each feature map may include multiple features, each feature corresponding to one or more elements of a corresponding frame of sensor data (e.g., pixels of an image or raw RADAR data). In one example, each set of feature maps generated by the feature extraction module 320 may include multiple feature maps with different spatial scales (or different resolutions), and such a set of feature maps is also referred to as a multi-scale feature map. Each feature map may have its own spatial resolution.

[0056] For example, the feature extraction module 320 may perform feature extraction on one or a frame of an image to generate a set of feature maps including one or more feature maps. Different feature maps in the set of feature maps may have different resolutions. Each feature of each feature map in the set of feature maps corresponds to one or more pixels of the image. Each row of features in each feature map in the set of feature maps corresponds to at least one row of pixels in the image. Each column of features in each feature map in the set of feature maps corresponds to at least one column of pixels in the image.

[0057] The feature update module 340 may include a correspondence mapping unit 341 and an attention layer 342 (also referred to as an attention unit or an attention neural network). The feature update module 340 is configured to receive feature maps (e.g., feature maps 325 and feature maps 335) as input, fuse or update the received feature maps, and output one or more fused (or updated) feature maps 345. In an example, the feature update module 340 may update the feature map 335 by using the feature map 325 received from the feature extraction module 320 to perform feature map fusion. The feature update module 340 may use an attention mechanism (e.g., a cross attention mechanism or a multi-head attention mechanism) to perform feature map updates and fusion.

[0058] The correspondence mapping unit 341 is configured to determine, for each feature in a feature graph (denoted as F1), one or more corresponding features in the feature graph F2 according to the position information of the feature and the position information of the feature in another feature graph (denoted as F2). In an example, the feature graph F1 is a feature graph in the feature graph 335, and the feature graph F2 is a feature graph in the feature graph 325. When the feature graph 325 includes multiple feature graphs, the correspondence mapping unit 341 selects a feature graph matching the feature graph F1 from the multiple feature graphs 325 as the feature graph F2.

[0059] In some embodiments, sensor data 315 includes data from, for example, Figure 1 The image obtained by the camera 121 shown, the feature map 325 generated by the feature extraction module 320 includes an image feature map, and the feature map 335 includes a BEV feature map. The correspondence mapping unit 341 can determine the corresponding one or more features in the image feature map for each feature in the BEV feature map according to the position information of the feature and the position information of the feature in the image feature map. In some embodiments, the feature map 325 includes multiple feature maps, and the correspondence mapping unit 341 can determine the corresponding one or more features in the matching image feature map for each feature in the BEV feature map according to the position information of the feature and the position information of the feature in the matching image feature map.

[0060] Typically, a camera (such as Figure 1A Cartesian coordinate system is established with the camera 121 shown as the origin (for example, with the optical center of the camera as the origin), also called the camera coordinate system. The three axes of the camera coordinate system are X, Y, and Z. The Z axis is parallel to the optical axis of the camera (i.e., the shooting direction of the camera lens). The X axis of the camera coordinate system is perpendicular to the Z axis and parallel to the ground, and the Y axis is perpendicular to the X axis and the Z axis. In an example, by properly setting the camera 121, the X axis and the Z axis of the camera coordinate system can be made parallel to the ground.

[0061] Typically, a two-dimensional pixel coordinate system can be established for each image captured by a camera (e.g., camera 121), with the origin of the pixel coordinate system being at a point on the image, such as the upper left corner of the image. The horizontal axis (also referred to as the u-axis) of the pixel coordinate system is parallel to the X-axis of the camera coordinate system. The vertical axis (also referred to as the v-axis) of the pixel coordinate system is parallel to the Y-axis of the camera coordinate system. The horizontal and vertical axes of the pixel coordinate system are parallel to the rows and columns of the image pixels, respectively. In other words, pixels with the same horizontal coordinate in the image form a column, and pixels with the same vertical coordinate form a row.

[0062] The correspondence mapping unit 341 can determine the coordinates or positions in the world coordinate system (i.e., the coordinate system established with a fixed point in the environment as the origin or pole) in the pixel coordinate system according to the calibration information of the camera 121. The correspondence mapping unit 341 can also determine the coordinates or positions in the coordinate system established with the sensor as the origin (or pole) in the pixel coordinate system according to the calibration information of the camera 121 and other sensors on the vehicle 120 (such as RADAR 122). Exemplarily, the camera calibration information may include the intrinsic and extrinsic parameters of the camera. The intrinsic parameters of the camera include the focal length, optical center position, etc. of the camera, and the extrinsic parameters include the position and orientation of the camera, etc. The calibration information of RADAR includes the posture of RADAR, such as the position and orientation of RADAR.

[0063] For example, Figure 1 Any point P in the environment 100 shown has coordinates (r bev ,φ bev , z), where r bev Represents the polar diameter, φ bev represents the polar angle, and z represents the coordinate of point P in the direction perpendicular to the ground. Point P has coordinates (x R ,y R , z RThe X-axis of the RADAR coordinate system is parallel to the orientation of the RADAR 122, the Y-axis is perpendicular to the X-axis and parallel to the ground, and the Z-axis is perpendicular to the X-axis and the Y-axis. By properly setting the RADAR 122, the X-axis and the Y-axis of the RADAR coordinate system can be made parallel to the ground. Point P has coordinates (x C ,y C , z C ), and at the same time, the coordinates of the point P in the polar coordinate system with RADAR 122 as the pole and the coordinates of the Cartesian coordinate system with RADAR 122 as the origin satisfy the following relationship:

[0064] x R =r bev cos(φ bev ) (Formula 1)

[0065] y R =r bev sin(φ bev ) (Formula 2)

[0066] z R =z (Formula 3)

[0067] The coordinates of point P in the Cartesian coordinate system with the camera as the origin (x C ,y C , z C ) can be expressed as:

[0068]

[0069] In Formula 4, R is a rotation matrix, and T is a translation vector, which is determined by the calibration information of the RADAR 122 and the calibration information of the camera 121 (eg, the external parameters of the camera), and are respectively expressed as:

[0070]

[0071]

[0072] The coordinates of point P in the Cartesian coordinate system with camera 121 as the origin and the coordinates of point P in the pixel coordinate system (i.e., the pixel coordinate system established for the image captured by camera 121) (x I ,y I ), the two satisfy the relationship:

[0073]

[0074]

[0075] The intersection of the camera optical axis and the image plane has coordinates (u 0, v 0 ) (i.e., the image principal point coordinates), which are determined by the optical center position of the camera. I is the horizontal coordinate of point P on the horizontal axis of the pixel coordinate system, y I is the ordinate of point P on the ordinate axis of the pixel coordinate system. x and f y The focal length of the camera 121 is related to the actual physical size of the pixels of the image captured by the camera 121 along the X-axis and Y-axis of the camera coordinate system. x , f y ,u 0 and v 0 It is the internal parameter of the camera and is related to the design technical indicators of the camera.

[0076] According to the above equations 1 to 6, the coordinates of point P in the polar coordinate system with RADAR 122 as the pole (r bev ,φ bev ) and the coordinates of point P in the pixel coordinate system (x I ,y I ) has the following relationship:

[0077]

[0078] As described above, the position of the grid cell corresponding to the feature in the BEV feature map in the environment 100 can be represented by the coordinates in the polar coordinate system (i.e., the RADAR polar coordinate system) with the RADAR 122 as the pole. bev ,φ bev ) indicates that, according to Formula 7, a feature on the BEV feature map can correspond to the feature map with coordinate x in the image. I A column of pixels corresponds to a column of features. In this way, the correspondence mapping unit 341 can map the features in the BEV feature map according to the position information of the feature (e.g., coordinates (r bev ,φ bev )) and the position information of the image feature map (or the matched image feature map) (e.g., coordinate x I ), determine the corresponding column of features in the image feature map.

[0079] In some embodiments, sensor data 315 includes Figure 1The original RADAR data obtained by the RADAR 122 shown, the feature map 325 generated by the feature extraction module 320 includes a RADAR feature map, and the feature map 335 includes a BEV feature map. The correspondence mapping unit 341 can determine the corresponding one or more features in the RADAR feature map for each feature in the BEV feature map according to the position information of the feature and the position information of the feature in the RADAR feature map. In some embodiments, the feature map 325 includes a plurality of feature maps, and the correspondence mapping unit 341 can determine the corresponding one or more features in the matched RADAR feature map for each feature in the BEV feature map according to the position information of the feature and the position information of the feature in the RADAR feature map matched therewith. The original RADAR data can be two-dimensional data in various different coordinate systems, one axis of these coordinate systems represents the distance (i.e., the distance relative to the RADAR), and this axis is also referred to as the distance axis in the present disclosure.

[0080] As described above, the position of the grid cell corresponding to the feature in the BEV feature map in the environment 100 can be represented by the coordinates in the polar coordinate system (i.e., the RADAR polar coordinate system) with RADAR 122 as the pole. bev ,φ bev ) indicates that a feature in the BEV feature map can correspond to a row of features in the RADAR feature map. This row of features corresponds to the original RADAR data with coordinates r R =r bev A row of pixels (the distance from the row of pixels to RADAR 122 is r R ). In this way, the correspondence mapping unit 341 can map the features in the BEV feature map according to the position information of the features (e.g., coordinates r bev ) and the position information of the RADAR feature map (or the matched RADAR feature map) (e.g., coordinates r R ), determine a row of features corresponding to the RADAR feature map. In one example, the pixels of the original RADAR data have the same size, and the coordinates of a pixel of the original RADAR data can be calculated based on the number of rows of pixels between the pixel and the origin of the range axis and the pixel size of the original RADAR data.

[0081] In some embodiments, the correspondence mapping unit 341 is configured to select, for each feature map 335, a feature map that matches it from the feature maps 325, and generate a feature map pair. In some embodiments, the correspondence mapping unit 341 is configured to select, for each feature map 335, a feature map that matches it from each group of feature maps 325, and generate a feature map pair. In some embodiments, the correspondence mapping unit 341 is configured to select, for each feature map 325, a feature map that matches it from the feature maps 335, and generate a feature map pair. In some embodiments, the correspondence mapping unit 341 is configured to select, for each feature map 325, a feature map that matches it from each group of feature maps 335, and generate a feature map pair.

[0082] In some embodiments, the feature maps in each feature map pair generated by the correspondence mapping unit 341 have the same resolution. In some embodiments, the feature maps in each feature map pair generated by the correspondence mapping unit 341 have the same dimension. In some embodiments, the feature maps in each feature map pair generated by the correspondence mapping unit 341 have the same resolution and the same dimension.

[0083] The attention layer 342 is configured to adopt an attention mechanism (e.g., a cross-attention mechanism or a multi-head attention mechanism) to update the feature map 325 using the feature map 335 or to update the feature map 335 using the feature map 325.

[0084] For example, the attention layer 342 can update each feature in each feature map 335 using one or more corresponding features in the feature map 325 determined by the correspondence mapping unit 341. Specifically, the attention layer 342 can use the features in the feature map 335 as query terms, the corresponding features in the feature map 325 as keywords and key values, and calculate the similarity between the query terms and each keyword as the weight of the corresponding key value. The attention layer 342 performs a weighted summation of all key values ​​using the obtained weights, and the features in the feature map 335 can be replaced by the features obtained by the summation. It should be noted that Figure 3 In the query, Q represents the query word, K represents the keyword, and V represents the key value.

[0085] In some embodiments, the attention layer 342 is configured to use an attention mechanism (e.g., a cross attention mechanism or a multi-head attention mechanism) to update another feature map using one feature map in the feature map pair determined by the correspondence mapping unit 341. For example, the attention layer 342 updates each feature in each feature map 335 using one or more corresponding features in the feature map 325 matched thereto determined by the correspondence mapping unit 341.

[0086] Sensor data 315 is Figure 1 When the image is captured by the camera 121 as shown, the process of performing the update operation by the feature update module 340 according to the exemplary embodiment of the present application is as follows: Figure 4 It should be noted that Figure 4 Zhongyu Figure 3 The same reference numerals represent the same contents. Figure 4 As shown, feature map 325 is an image feature map, and feature map 335 is a BEV feature map. Figure 4 Also shown are the u and v axes of the pixel coordinate system and the polar axis ρ of the polar coordinate system with RADAR 122 as the pole. The attention layer 342 takes a feature of the BEV feature map 335, such as feature 465 (denoted as f(r bev ,φ bev )) as the query word. (r bev ,φ bev ) is the coordinate of the grid cell corresponding to the feature 465 in the polar coordinate system with the RADAR 122 as the pole. According to the calibration parameters of the camera 121 and the calibration parameters of the RADAR 122, the correspondence mapping unit 341 determines the coordinate (r bev ,φ bev ) corresponds to the image with horizontal coordinate x I A column of pixels, which corresponds to a column of features 426 in the feature map 325 (expressed as feature {g(x I ,y k )} k＝1，2，...H , where y k is the vertical coordinate of the pixel, H represents the total number of rows of the feature map 325), so that the correspondence mapping unit 341 can determine the feature f(r bev ,φ bev ) corresponds to a column of features 426 in feature map 325. The feature {g(x I ,y k )} k＝1，2，...H As the characteristic f(r bev ,φ bev ) corresponding to the keyword (key) and key value (value). The attention layer 342 can calculate the query word f(r bev ,φ bev ) with each corresponding keyword as the weight of the corresponding key value. For example, the similarity between the query word and the corresponding keyword can be obtained by the vector dot product of the query word and each keyword. bev ,φ bev ) is weighted summed up for all key values ​​corresponding to the feature Use features Replace the feature f(r bev ,φ bev), complete the feature f(r bev ,φ bev ). The attention layer 342 can update each feature in the feature map 335 in the same way to update the feature map and obtain the feature map 345.

[0087] The sensor data 315 is Figure 1 When the original RADAR data (eg, RT data) obtained by the RADAR 122 shown in FIG. 1 is obtained, the process of performing the update operation by the feature update module 340 according to the exemplary embodiment of the present application is as follows: Figure 5 The following table shows the instructions for use. Figure 5 Zhongyu Figure 3 The same reference numerals denote the same contents.

[0088] RT data, i.e., Range-Time data, is the result of a Fast Fourier Transformation (FFT) of the RADAR analog-to-digital converter (ADC) data in the distance dimension. Usually, RT data is a complex tensor of size m×n×s, where m represents the number of samples of an intermediate frequency pulse signal, n represents the number of intermediate frequency pulse signals contained in a frame of data, and s represents the product of the number of antenna transmitting units and receiving units. m, n, and s are all integers greater than 1. As an example, m, n, and s are 512, 256, and 192, respectively.

[0089] like Figure 5 As shown, feature map 325 is a RADAR feature map, and feature map 335 is a BEV feature map. Figure 5 Also shown are two mutually perpendicular coordinate axes, T and R, of the RT data, where the T axis represents time and the R axis represents the distance relative to RADAR. Figure 5 The polar axis ρ of the polar coordinate system with RADAR 122 as the pole is also shown. The attention layer 342 takes a feature of the BEV feature map 335, such as feature 565 (denoted as f(r bev ,φ bev )) as the query word. (r bev ,φ bev ) is the coordinate of the grid cell corresponding to feature 565 in the polar coordinate system with RADAR 122 as the pole. The correspondence mapping unit 341 determines the coordinate (r bev ,φ bev ) corresponds to a row of pixels with coordinates r (coordinates on the R axis) in the RT data, and this row of pixels corresponds to a row of features 528 (expressed as features {h(r R , tn )} n＝1，2，...N , where t n is the coordinate on the T axis, and N represents the total number of columns of the feature map 325). In this way, the correspondence mapping unit 341 can determine the feature f(r bev ,φ bev ) corresponds to a row of features 528 in feature map 325. The feature {h(r R , t n )} n＝1，2，...N As the characteristic f(r bev ,φ bev ) corresponding to the keyword (key) and key value (value). The attention layer 342 can calculate the query word f(r bev ,φ bev ) with each corresponding keyword as the weight of the corresponding key value. For example, the similarity between the query word and the corresponding keyword can be obtained by the vector dot product of the query word and each keyword. bev ,φ bev ) is weighted summed up for all key values ​​corresponding to the feature Use features Replace the feature f(r bev ,φ bev ), complete the feature f(r bev ,φ bev ). The attention layer 342 can update each feature in the feature map 335 in the same way to update the feature map and obtain the feature map 345.

[0090] In some embodiments, the feature update module 340 further includes a first residual and normalization (Add&Norm) layer ( Figure 3 ), the first residual and normalization layer combines the output of the attention layer 342 with the input of the attention layer 342 and applies normalization to the combined result.

[0091] In some embodiments, the feature update module 340 further includes a feed-forward layer (not shown). When the feature update module 340 includes a first residual and a normalization layer, the input received by the feed-forward layer may be the output of the first residual and normalization layer. When the feature update module 340 does not include a first residual and a layer normalization layer, the input received by the feed-forward layer may be the output of the attention layer 342.

[0092] In some embodiments, the feature update module 340 further includes a second residual and normalization layer ( Figure 3 (not shown in the figure), the second residual and normalization layer combines the output of the feed-forward layer with the input of the feed-forward layer and applies normalization to the combined result.

[0093] Fig. 6A 6 is a schematic diagram of a data processing system 600 according to an example embodiment of the present application. An example of a data processing system 600 is a computer program implemented on one or more computing devices at one or more locations. In one example, the data processing system 600 is a computer program implemented on one or more computing devices at one or more locations. Figure 1 The computing device 130 shown or Figure 2 The computer program implemented on the computing device 200 shown in FIG. 6 is a computer program implemented on the computing device 200 shown in FIG. 6. The data processing system 600 can fuse sensor data obtained from sensors of different perspectives (ie, sensor data of different modalities) using a unified representation space (eg, BEV space).

[0094] The data processing system 600 may include a data processing device 630 and a data processing device 640. The data processing system 600 may receive data from one or more sensors (e.g. Figure 1 The sensor data 615 and 616 are received as input by the vehicle 120 as shown in FIG. 1 . In one example, the sensor data 615 and 616 include sensor data obtained from different types of sensors. In one example, the sensor data 615 may include Figure 1 The sensor data 616 may include images obtained by the camera 121. Figure 1 In another example, the sensor data 616 may include RADAR data obtained from the RADAR 122 shown in FIG. Figure 1 The sensor data 615 may include images obtained by the camera 121. Figure 1 RADAR data acquired by RADAR122 is shown.

[0095] The image obtained by the data processing system 600 from the camera 121 may include any suitable type of image, for example, a two-dimensional image with multiple channels (eg, an RGB image). The RADAR data obtained by the data processing system 600 from the RADAR 122 may be raw RADAR data.

[0096] Usually, the raw RADAR data obtained by the data processing system from the RADAR contains information such as the distance and orientation of objects in the environment relative to the RADAR, and may also include information such as the speed of objects in the environment. The raw RADAR data includes at least one of RAD (range-azimuth-Doppler) tensor data, RT (range-time) data, RA (range-azimuth) data, and RD (range-Doppler) data. The raw RADAR data can be two-dimensional data, in which one axis represents the distance (i.e., the distance relative to RADAR 122). Different raw RADAR data have different contents represented by the other axis. For example, the other axis of RT data represents time, and the other axis of RD data represents speed. RA and RAD data are usually represented by polar coordinates, in which the polar axis represents the distance (i.e., the distance relative to RADAR 122).

[0097] like Fig. 6A As shown, the data processing device 630 may include a feature extraction module 632 and a feature update module 634. The data processing device 630 may receive the sensor data 615 and the feature map 625 as inputs, and generate a feature map 635 as output.

[0098] The feature extraction module 632 can perform feature extraction on the sensor data 615 to generate one or more feature maps to provide to the feature update module 634. The feature update module 634 can include a correspondence mapping unit and an attention layer ( Fig. 6A 634). The feature update module 634 is configured to receive a feature map as input, fuse or update the received feature map, and output one or more fused (or updated) feature maps 635. In one example, the feature update module 634 can perform feature map fusion by updating the feature map 625 using the feature map output from the feature extraction module 632. The feature update module 634 can use an attention mechanism (such as a cross attention mechanism or a multi-head attention mechanism) to perform feature map update and fusion.

[0099] like Fig. 6A As shown, the data processing device 640 may include a feature extraction module 642 and a feature update module 644. The data processing device 640 may receive the sensor data 616 and the feature map 635 as inputs, and generate a feature map 645 as output.

[0100] The feature extraction module 642 can perform feature extraction on the sensor data 616 to generate one or more feature maps to provide to the feature update module 644. The feature update module 644 can include a correspondence mapping unit and an attention layer ( Fig. 6A644 is configured to receive a feature map as input, fuse or update the received feature map, and output one or more fused (or updated) feature maps 645. In one example, the feature update module 644 can update the feature map 635 by using the feature map output from the feature extraction module 642 to perform feature map fusion. The feature update module 644 can use an attention mechanism (such as a cross attention mechanism or a multi-head attention mechanism) to perform feature map update and fusion.

[0101] The data processing devices 630 and 640 may have substantially the same structure and function. The examples of the data processing devices 630 and 640 are both referenced. Figure 3 Accordingly, the feature extraction module 632 and the feature extraction module 642 may have substantially the same structure and function, and the examples of the feature extraction module 632 and the feature extraction module 642 are all referenced Figure 3 The feature extraction module 320 described above; the feature update module 634 and the feature update module 644 may have substantially the same structure and function, and the examples of the feature update module 634 and the feature update module 644 are all referenced Figure 3 Described feature update module 340.

[0102] In some embodiments, the neural networks in data processing devices 630 and 640 are both learned neural networks. The neural network in data processing device 630 can be trained using sensor data of the same type as sensor data 615, and the neural network in data processing device 640 can be trained using sensor data of the same type as sensor data 616.

[0103] In some embodiments, the data processing system 600 may include an initialization module 620 configured to generate one or more initialized feature maps 625 (ie, a set of feature maps). The initialization module 620 may include a space division unit 621 and a position encoding unit 622 .

[0104] The space division unit 621 is configured to perform space division. For example, the space division unit 621 may Figure 1 The predetermined space in the environment 100 is shown divided into a grid. The predetermined space may be a space within a field of view of a sensor (eg, RADAR 122) of the vehicle 120 and within a predetermined range from the sensor.

[0105] The predetermined space in the environment 100 may be a bird's-eye view space. For the bird's-eye view space, the space division unit 621 may divide a predetermined range of a plane parallel to the ground into grids to obtain a BEV grid. The grid units may be arranged in a two-dimensional pattern. Each grid unit is regarded as an infinitely high pillar. For example, the space division unit 621 may be arranged in a two-dimensional pattern. Figure 1 The fixed points in the environment 100 are shown as poles, or Figure 1 The vehicle 120 or the sensor on the vehicle (e.g., RADAR 122) in the environment 100 shown is a pole, and a ray parallel to the ground is drawn from the pole as a polar axis, and a polar coordinate system is established on the plane where the ray is located (e.g., a plane parallel to the ground). The space division unit 621 can divide the predetermined range of the plane where the ray is located (e.g., within the field of view of RADAR 122 and within the predetermined range of RADAR 122) into grids to obtain a BEV grid. The position of each grid cell of the BEV grid in the environment 100 can be represented by the coordinates (r bev ,φ bev ). Among them, r bev represents the polar diameter, that is, the distance between the grid unit and the pole; φ bev Represents the polar angle, that is, the angle between the line segment from the pole to the grid cell and the polar axis. In one example, each grid cell of the BEV grid has the same size as the pixel of the original RADAR data. In one example, the number of grid cell rows of the BEV grid is the same as the number of pixel rows of the original RADAR data. In one example, the grid cells of the BEV grid are arranged in a two-dimensional array in rows and columns, the grid cells of the BEV grid have the same size, and the coordinates of a grid cell of the BEV grid can be calculated based on the number of rows and columns of the grid cells between the grid cell and the pole and the size of the grid cell.

[0106] The position encoding unit 622 is configured to generate one or more feature maps 625, also called initialized feature maps, by position encoding. For example, the position encoding unit 622 performs position encoding on the grid divided by the space division unit 621, thereby generating one or more feature maps 625. Each feature map 625 may include multiple features. An example of a feature map 625 is Figure 3 Feature diagram 335 is shown.

[0107] In some embodiments, the feature map 625 generated by the position encoding unit 622 may include one or more sets of feature maps. Each set of feature maps may include one or more feature maps. Each feature map may include multiple features. In one example, each set of feature maps may include multiple feature maps with different spatial scales (or different resolutions), thereby forming a multi-scale feature map. Each feature map may have its own spatial resolution.

[0108] In some embodiments, the position encoding unit 622 may perform position encoding for each grid cell according to the relative position of the grid cell within the grid (i.e., the position of the grid cell relative to other grid cells), and generate a corresponding feature for the grid cell. That is, the position encoding unit 622 may generate a feature for one grid cell. In another example, the position encoding unit 622 may perform position encoding for every two or three grid cells according to the relative positions of these grid cells within the grid (i.e., the positions of these grid cells relative to other grid cells), and generate a corresponding feature for these grid cells. That is, each feature may correspond to one or more grid cells. The feature map generated by the position encoding unit 622 may have different resolutions. Each feature of the feature map has position information to reflect the position of the grid cell corresponding to the feature.

[0109] In some embodiments, the position encoding unit 622 may perform position encoding on the BEV grid generated by the space division unit 621 to generate a BEV feature map. Each feature in the BEV feature map may correspond to one or more cells of the BEV grid. As described above, the position of each grid cell of the BEV grid generated by the space division unit 621 may be represented by a coordinate in a polar coordinate system (r bev ,φ bev ), so each feature in the BEV feature map has position information (r bev ,φ bev ) to represent the coordinates of the grid unit corresponding to the feature. In this way, the features in the BEV feature map can be represented by f i (r bev ,φ bev ) indicates that when a feature corresponds to multiple grid cells, the coordinate (r bev ,φ bev ) represents the position of any one of the multiple grid cells corresponding to the feature.

[0110] According to the data processing system disclosed in the present invention, different sensor data can be fused, and in particular, sensor data with limited dimensions (e.g., one dimension) such as raw RADAR data can be fused with other sensor data, such as raw RADAR data and image data. The fusion of raw RADAR data and image data can achieve a positioning accuracy close to that of a laser radar, and can accurately detect the spatial position of objects in the environment. When the data processing system disclosed in the present invention is applied to an autonomous driving vehicle, the safety performance of the vehicle in bad weather can be greatly improved.

[0111] Figure 6B is a schematic diagram of a data processing system according to another exemplary embodiment of the present application. Figure 6BThe data processing system 610 is shown with Fig. 6A The difference between the data processing system 600 shown is that the data processing system 610 also includes a task execution module 650. It should be noted that: Figure 6B Zhongyu Fig. 6A The same contents are indicated by the same reference numerals. Figure 6B and Fig. 6A The difference.

[0112] like Figure 6B As shown, the data processing system 610 may include a task execution module 650, which is used to use the feature map 645 output by the feature update module 644 to execute one or more computer tasks and obtain a task execution result 655. For example, the task execution module 650 can predict the target position and category confidence corresponding to each query unit in the target query using the feature map 645 according to the target query, and output the prediction result as the task execution result 655. The task execution module 650 may include a decoder, which executes one or more computer tasks according to the feature map 645. An example of a decoder is a transformer decoder. Those skilled in the art will appreciate that the decoder may also be any other decoder or neural network known in the art.

[0113] The computer tasks performed by the task execution module 650 may include any of a variety of tasks. For example, the tasks may include one or more sensor data classification tasks. In the sensor data classification task, the output generated by the task execution module 650 is a score for each of a set of object categories, each score representing the estimated likelihood that the sensor data (e.g., sensor data 615 and sensor data 616) contains an object belonging to that category. As another example, the task may include one or more target detection tasks (e.g., 3D target detection tasks associated with autonomous driving). In the target detection task, the output generated by the task execution module 650 includes a bounding box or other symbol that identifies the location and size of a specific type of object in the sensor data (e.g., sensor data 615 and sensor data 616). The task execution module 650 can perform target detection at different distance ranges for feature maps of different scales.

[0114] Figure 7 6 is a schematic diagram of an example of a task execution module 650. The task execution module includes a transformer decoder and is configured to perform a target detection task according to a feature map and output a target detection result. Figure 7As shown, the transformer decoder includes a decoding layer, and the number of decoding layers can be one or more. The decoding layer includes a multi-head attention layer and a multi-scale deformation attention layer. The multi-head attention can be self-attention, and the multi-scale deformation attention can be cross attention. The object query is combined with the position encoding to obtain query terms, keywords and key values ​​as inputs to the multi-head attention layer. The first residual and normalization layer combines the output of the multi-head attention layer with the input of the multi-head attention layer, and applies normalization to the combined result. It should be noted that Figure 7 In the query, Q represents the query word, K represents the keyword, and V represents the key value.

[0115] The output of the first residual and normalization layer is combined with the positional encoding as the query term (Q) and fed into the multi-scale deformation attention layer, and the feature map (e.g. Figure 6B The features in the feature map 645 output by the data processing device 640 are also input into the multi-scale deformation attention layer as keywords (K) and key values ​​(V), and the output of the multi-scale deformation attention layer is obtained. After the target query is combined with the position encoding, a reference point representing the target position can also be obtained as the input of the multi-scale deformation attention layer.

[0116] The second residual and normalization layer combines the output of the multi-scale deformation attention layer with the input of the multi-scale deformation attention layer, and applies normalization to the combined result. The output of the second residual and normalization layer is the input of the feedforward layer. The third residual and normalization layer combines the output of the feedforward layer with the input of the feedforward layer, and applies normalization to the combined result. The output of the third residual and normalization layer passes through the first fully connected layer to obtain the category score of the target, and passes through the second fully connected layer to obtain the offset of the target position, thereby obtaining the target detection result. An example of the offset of the target position is the correction amount of the position, size and rotation angle of the target bounding box in polar coordinates. The reference point can be updated according to the offset of the target position.

[0117] In some embodiments, the classification head (i.e., the first fully connected layer) of the task execution module 650 can be trained using focal loss, and the smooth L 1 The regression head (i.e., the second fully connected layer) of the loss training task execution module 650 can realize the training of the neural network of the entire data processing system because the loss will be back-propagated to the entire data processing system.

[0118] Figure 8 is a flowchart of a data processing method 700 according to an exemplary embodiment of the present application. The method can be performed by Figure 3 The data processing device 300 shown or Fig. 6A The data processing system 600 or Figure 6BThe data processing system 610 is shown executing.

[0119] like Figure 8 As shown, in step 720, sensor data about the environment collected by the sensor is received. For example, Figure 3 The feature extraction module 320 or Fig. 6A and 6B The illustrated feature extraction module 632 or 642 receives sensor data from one or more sensors.

[0120] In step 740, a first feature map is generated based on the sensor data. Figure 3 The feature extraction module 320 or Fig. 6A and 6B The illustrated feature extraction module 632 or 642 performs feature extraction on the sensor data to generate one or more feature maps as a first feature map.

[0121] In step 760, a second feature map for describing the environment is updated according to the first feature map. Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature updating module 634 or 644 shown updates the second feature map according to the first feature map. Each feature in the first feature map and the second feature map has position information for indicating the position in the environment corresponding to the feature.

[0122] In some embodiments, Figure 8 The illustrated step 760 may be implemented via steps 761 and 762 .

[0123] Figure 8 As shown, in step 761, by Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature updating module 634 or 644 shown determines multiple features in the first feature map for each feature of at least one feature in the second feature map according to the position information of the feature in the second feature map and the position information of the feature in the first feature map.

[0124] In step 762, by Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature updating module 634 or 644 shown updates corresponding features in the second feature map according to the determined multiple features in the first feature map.

[0125] Fig. 9 is a flowchart of a data processing method 800 according to another exemplary embodiment of the present application. The method may be performed by Figure 3 The data processing device 300 shown or Fig. 6A The data processing system 600 or Figure 6B The data processing system 610 is shown executing.

[0126] like Fig. 9 As shown, in step 820, first sensor data about the environment collected by the first sensor is received. Figure 3 The feature extraction module 320 or Fig. 6A and 6B The feature extraction module 632 or 642 shown receives first sensor data from one or more sensors. The first sensor data may include Figure 1 The RADAR 122 shown acquires RADAR data (eg, raw RADAR data). The raw RADAR data includes at least one of RAD data, RT data, RA data, and RD data.

[0127] In step 840, a first feature map is generated based on the first sensor data. Figure 3 The feature extraction module 320 or Fig. 6A and 6B The feature extraction module 632 or 642 shown performs feature extraction on the first sensor data to generate one or more feature maps as the first feature map. Figure 3 The illustrated feature extraction module 320 may perform feature extraction on the first sensor data to generate a feature map as a first feature map.

[0128] Each feature in the first feature map has position information indicating the location in the environment to which the feature corresponds. In one example, the first feature map is a RADAR feature map (e.g., an original RADAR feature map). The features in the RADAR feature map have position information indicating the environment to which the feature corresponds (e.g., Figure 1 In one example, the position information of each feature in the RADAR feature map includes information about the distance relative to the RADAR (i.e., the distance from the location in the environment corresponding to the feature to the RADAR). In another example, the position information of each feature in the RADAR feature map includes information about the distance and orientation relative to the RADAR (i.e., the distance and orientation from the location in the environment corresponding to the feature to the RADAR).

[0129] In some embodiments, the first sensor data received by the feature extraction module may include one or more frames of sensor data (e.g., or one or more frames of raw RADAR data). The first feature map may include one or more groups of feature maps. Each group of feature maps may correspond to one frame of sensor data. Each group of feature maps may include one or more feature maps. Each feature map may include multiple features, each feature corresponding to one or more elements of a corresponding frame of sensor data (e.g., pixels of raw RADAR data). In one example, each group of feature maps may include multiple feature maps with different spatial scales (or different resolutions), thereby forming a multi-scale feature map.

[0130] In step 860, a second feature map for describing the environment is updated according to the first feature map. Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature updating module 634 or 644 shown updates the second feature map according to the first feature map. Each feature in the second feature map has position information for indicating the location of the environment corresponding to the feature. In an example, each feature in the second feature map has a grid unit corresponding to the feature in the environment (e.g. Figure 1 The second feature map may be a BEV feature map. The position information of each feature in the BEV feature map includes coordinates in a polar coordinate system with RADAR as the pole.

[0131] An example of the second feature map is Fig. 6A and 6B As described above, the space division unit 621 can divide the environment (e.g. Figure 1 The predetermined space in the environment 100 shown in the figure is divided into grids, and the position encoding unit 622 can perform position encoding on the grid cells of the grid generated by the space division unit 621, thereby generating a feature map 625. In this way, the feature map 625 is related to the environment 100, and it can also be said that the feature map 625 reflects or describes the environment 100. In an example, the second feature map is Figure 3 The feature map 335 shown is similar to the feature map 625, and the feature map 335 is also related to the environment. In one example, the second feature map is Fig. 6A and 6B As described above, the data processing device 630 can process the data from the sensor (e.g. Figure 1 The sensor data received by the sensor of the vehicle 120 as shown in the figure is used to update the feature map 625 to obtain the feature map 635, so that the feature map 635 includes the sensor data associated with the environment 100 around the vehicle 120, so that the feature map 635 is also related to the environment 100.

[0132] In some embodiments, the second feature map includes one or more sets of feature maps. Each set of feature maps may include multiple feature maps with different spatial scales (or different resolutions), thereby forming a multi-scale feature map. Wherein, each feature in the first feature map and the second feature map has position information for indicating the position in the environment corresponding to the feature.

[0133] In some embodiments, Fig. 9 Step 860 shown can be implemented by steps 861 and 862. Fig. 9 As shown, in step 861, by Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature updating module 634 or 644 (specifically, the corresponding relationship mapping unit of these modules) selects or determines multiple features in the first feature map for each feature of at least one feature in the second feature map according to the position information of the feature in the second feature map and the position information of the feature in the first feature map. Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature update module 634 or 644 shown selects or determines, for each feature of at least one feature in the second feature map, a plurality of features in the first feature map corresponding to the position in the environment corresponding to the feature according to the position information of the feature in the second feature map. When the sensor data received in step 820 is RADAR data, Figure 3 The feature update module 340 shown or Fig. 6A and 6B The position information of the multiple features in the first feature map determined by the feature updating module 634 or 644 shown indicates the same distance relative to the RADAR (for example, the distances of the pixels corresponding to these features relative to the RADAR are the same).

[0134] In step 862, by Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature update module 634 or 644 shown (specifically, the attention layer of these modules) updates the corresponding features in the second feature map based on the multiple features determined in the first feature map.

[0135] In some embodiments, Figure 3 The feature update module 340 shown or Fig. 6A and 6BThe feature update module 634 or 644 shown calculates the similarity between the feature in the second feature map and each of the multiple features selected or determined in the first feature map through the attention mechanism, for each feature in the at least one feature in the second feature map, wherein the feature in the second feature map is used as the query value in the attention mechanism, and each of the multiple features in the first feature map is used as the keyword and key value in the attention mechanism. Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature updating module 634 or 644 shown uses the similarity as a weight to perform weighted summation on the key values, and updates the query value according to the summation result.

[0136] In some embodiments, Fig. 9 The data processing method 800 shown in the figure also includes steps 870, 880 and 890. In step 870, second sensor data about the environment collected by the second sensor is received. Figure 3 The feature extraction module 320 or Fig. 6A and 6B The feature extraction module 632 or 642 shown receives second sensor data from one or more sensors. In some embodiments, the second sensor data and the first sensor data are sensor data collected by different types of sensors. The second sensor data may include Figure 1 An image obtained by the camera 121 is shown. The image obtained from the camera 121 may include any suitable type of image, for example, a two-dimensional image having multiple channels (eg, an RGB image).

[0137] In step 880, a third feature map is generated based on the second sensor data. Figure 3 The feature extraction module 320 or Fig. 6A and 6B The feature extraction module 632 or 642 shown performs feature extraction on the second sensor data to generate one or more feature maps as the third feature map. Figure 3 The illustrated feature extraction module 320 may perform feature extraction on the second sensor data to generate a feature map as a third feature map.

[0138] Each feature in the third feature map has position information for indicating the position in the second sensor data (eg, image) corresponding to the feature. The features in the third feature map are arranged in a two-dimensional array in rows and columns.

[0139] In some embodiments, the second sensor data received by the feature extraction module may include one or more frames of sensor data (e.g., or one or more frames of images). The third feature map may include one or more groups of feature maps. Each group of feature maps may correspond to one frame of sensor data. Each group of feature maps may include one or more feature maps. Each feature map may include multiple features, each feature corresponding to one or more elements of a corresponding frame of sensor data (e.g., pixels of an image). In one example, each group of feature maps may include multiple feature maps with different spatial scales (or different resolutions), thereby forming a multi-scale feature map. In step 890, the second feature map used to describe the environment is updated according to the third feature map. For example, it can be Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature updating module 634 or 644 shown updates the second feature map according to the third feature map.

[0140] In some embodiments, Fig. 9 Step 890 shown is implemented by steps 891 and 892. Fig. 9 As shown, in step 891, by Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature updating module 634 or 644 (specifically, the corresponding relationship mapping unit of these modules) selects or determines multiple features in the third feature map for each feature of at least one feature in the second feature map according to the position information of the feature in the second feature map and the position information of the feature in the third feature map. Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature update module 634 or 644 shown selects or determines, for each feature of at least one feature in the second feature map, multiple features in the third feature map corresponding to the position in the environment corresponding to the feature according to the position information of the feature in the second feature map.

[0141] In some embodiments, the first sensor data received in step 820 is RADAR data received from RADAR, the second sensor data received in step 870 is an image received from a camera, and the features in the third feature map are arranged in a two-dimensional array in rows and columns. Figure 3 The feature update module 340 shown or Fig. 6A and 6BThe feature update module 634 or 644 shown determines multiple positions in the two-dimensional array based on the position information of the feature in the second feature map and the calibration parameters of the camera and RADAR for each feature in at least one feature in the second feature map, and determines multiple features in the third feature map based on the multiple positions in the two-dimensional array. In an example, the multiple features of the determined third feature map include each feature in at least one column of features in the third feature map.

[0142] In step 892, by Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature update module 634 or 644 shown (specifically, the attention layer of these modules) updates the corresponding features in the second feature map based on the multiple features in the determined third feature map.

[0143] In some embodiments, Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature update module 634 or 644 shown calculates the similarity between the feature in the second feature map and each of the multiple features selected or determined in the third feature map through the attention mechanism, for each feature in the at least one feature in the second feature map, wherein the feature in the second feature map is used as the query value in the attention mechanism, and each of the multiple features in the third feature map is used as the keyword and key value in the attention mechanism. Figure 3 The feature update module 340 shown or Fig. 6A and 6B The feature update module 634 or 644 shown uses the similarity as a weight to perform a weighted summation on the key value, and updates the query value according to the summation result. Fig. 9 The order in which the steps are performed. For example Fig. 9 Steps 820, 840, and 860 may be executed before steps 870, 880, and 890, or after steps 870, 880, and 890, or steps 820, 840, and 860 may be executed in parallel with steps 870, 880, and 890.

[0144] It should be noted that in Fig. 9 Steps 820, 840, and 860 in Fig. 6A and 6B When executed by components of data processing device 630 as shown, steps 870, 880, and 890 may be executed by components of data processing device 640. Fig. 6A and 6BWhen the components of the data processing device 640 shown in FIG. 8 are executed, steps 870, 880, and 890 may be executed by components of the data processing device 630. In some embodiments, the method 800 further includes executing the target task according to the updated second feature map (i.e., the second feature map updated by steps 860 and 890). Figure 6B The task execution module 650 shown executes the target task according to the updated second feature map.

[0145] In some embodiments, the method 800 further includes performing initialization by position encoding to generate the second feature map. Fig. 6A and 6B The initialization module 620 shown performs initialization to generate the second feature map.

[0146] According to the data processing method disclosed in the present invention, a feature map of a specific representation space (such as a BEV space) can be generated based on the sensor data received from the sensor, so that the sensor data of different perspectives can be projected into a unified representation space, especially the sensor data with limited dimensions (such as one dimension) such as the original RADAR data can be projected into the unified representation space. In addition, according to the data processing method disclosed in the present invention, different sensor data can be fused, especially the sensor data with limited dimensions (such as one dimension) such as the original RADAR data can be fused with other sensor data, such as fusing the original RADAR data with image data. The fusion of the original RADAR data and the image data can achieve a positioning accuracy close to that of a laser radar, and can accurately detect the spatial position of objects in the environment. When the data processing system disclosed in the present invention is applied to an autonomous driving vehicle, it can greatly improve the safety performance of the vehicle in bad weather.

[0147] Although this paper includes many details, these details should not be interpreted as limitations to the scope of the claimed invention, but are interpreted as descriptions of features peculiar to specific embodiments. Some features described in the context of different embodiments herein may also be combined in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be separated or implemented in multiple embodiments with any suitable sub-combination. In addition, although features may be described as acting with some combinations and even initially claimed above, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may be directed to a variation of a sub-combination or a sub-combination. Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring to perform such operations in the particular order shown or in a sequential order, or to perform all described operations to achieve desired results.

[0148] Note that the above are only preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure is described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the attached claims.

Claims

1. A data processing method, comprising: receiving sensor data about the environment collected by the sensor; generating a first feature map according to the sensor data; updating a second feature map for describing the environment according to the first feature map, Each feature in the first feature map and the second feature map has position information, which is used to indicate the position in the environment corresponding to the feature. Wherein, updating the second feature map used to describe the environment according to the first feature map includes: For each feature in at least one feature in the second feature map: Determine a plurality of features in the first feature map according to the position information of the feature in the second feature map and the position information of the feature in the first feature map; The feature in the second feature map is updated according to the determined multiple features in the first feature map.

2. The method according to claim 1, wherein: The sensor is a first sensor, the sensor data is first sensor data, and the method further comprises: receiving second sensor data about the environment collected by a second sensor; generating a third feature map according to the second sensor data, wherein each feature in the third feature map has position information for indicating a position in the second sensor data corresponding to the feature; A second feature map for describing the environment is updated according to the third feature map.

3. The method according to claim 2, wherein: Updating the second feature map for describing the environment according to the third feature map includes: For each feature of the at least one feature in the second feature map: Determine a plurality of features in the third feature map according to the position information of the feature in the second feature map and the position information of the feature in the third feature map; The feature in the second feature map is updated according to the determined multiple features in the third feature map.

4. The method according to claim 3, wherein: The first sensor comprises RADAR, the first sensor data comprises RADAR data, the second sensor comprises a camera, and the second sensor data comprises an image.

5. The method according to claim 4, wherein: The features in the third feature map are arranged into a two-dimensional array in rows and columns. Wherein, according to the position information of the feature in the second feature map and the position information of the feature in the third feature map, determining multiple features in the third feature map includes: According to the position information of the feature in the second feature map, multiple positions in the two-dimensional array are determined using calibration parameters of the camera and the RADAR, A plurality of features in the third feature map is determined according to a plurality of locations in the two-dimensional array.

6. The method according to claim 5, wherein: The plurality of features of the third feature map includes each feature in at least one column of features in the third feature map.

7. The method according to claim 3, wherein: Updating the feature in the second feature map according to the determined multiple features of the third feature map includes: Through the attention mechanism, the similarity between the feature in the second feature map and each of the multiple features in the third feature map is calculated, wherein the feature in the second feature map is used as the query value in the attention mechanism, and each of the multiple features in the third feature map is used as the keyword and key value in the attention mechanism. Taking the similarity as the weight, weighted sum is performed on the key values, The feature in the second feature map is updated according to the summation result.

8. The method according to claim 2, wherein: Updating the second feature map for describing the environment according to the third feature map is performed before updating the second feature map for describing the environment according to the first feature map.

9. The method of claim 2, wherein the first, second, and third feature maps each comprise a multi-scale feature map having different resolutions.

10. The method according to any one of claims 1 to 9, wherein: Updating the feature in the second feature map according to the determined multiple features in the first feature map includes: Through the attention mechanism, the similarity between the feature in the second feature map and each of the multiple features in the first feature map is calculated, wherein the feature in the second feature map is used as the query value in the attention mechanism, and each of the multiple features in the first feature map is used as the keyword and key value in the attention mechanism. Taking the similarity as the weight, weighted sum is performed on the key values, The feature in the second feature map is updated according to the summation result.

11. The method according to any one of claims 1 to 9, wherein: The sensor includes a RADAR, the first feature map includes an original RADAR feature map, and the location in the environment indicated by the location information of the features in the first feature map includes a distance relative to the RADAR.

12. The method according to claim 11, wherein: The distances relative to the RADAR indicated by the position information of the plurality of features in the first feature map are the same as each other.

13. The method according to any one of claims 1 to 9, wherein: The sensor data includes raw RADAR data, and the raw RADAR data includes at least one of RAD data, RT data, RA data, and RD data.

14. The method according to any one of claims 1 to 9, wherein: The second feature map includes a BEV feature map, and the position information of each feature in the second feature map includes coordinates in a polar coordinate system with RADAR as a pole.

15. The method according to any one of claims 1 to 9, further comprising: Initialization is performed by position encoding to generate the second feature map.

16. The method according to any one of claims 1 to 9, further comprising: According to the second feature map, a target task is performed.

17. An apparatus for data processing, comprising: processor; and A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 16.

18. A computer-readable storage medium storing a program, the program comprising instructions, which, when executed by one or more processors of a computing device, cause the computing device to perform the method according to any one of claims 1 to 16.

Images