Image processing method and apparatus

Abstract

The application provides an image processing method and device. The method comprises the following steps: obtaining a bird's-eye view feature map of a plurality of continuously generated images and corresponding positioning information; for each bird's-eye view feature map in the plurality of bird's-eye view feature maps, the following operations are performed: rotating the bird's-eye view feature map based on an orientation angle of an autonomous vehicle corresponding to the bird's-eye view feature map to obtain a rotated bird's-eye view feature map; determining a target region occupied by the rotated bird's-eye view feature map in a fusion feature map according to a position coordinate of the autonomous vehicle corresponding to the bird's-eye view feature map; and fusing the rotated bird's-eye view feature map into the fusion feature map based on the target region. Each of the plurality of continuously generated bird's-eye view feature maps is processed respectively, thereby shortening the time delay required for image processing. Meanwhile, the plurality of continuously generated bird's-eye view feature maps are fused into the fusion feature map in a time sequence, thereby obtaining a fusion feature map of the plurality of continuously generated bird's-eye view feature maps, and improving the accuracy of the prediction of the surrounding environment of the autonomous vehicle.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an image processing method and apparatus. Background Technology

[0002] In autonomous driving, utilizing the temporal information contained in multi-frame bird's-eye view feature maps is particularly important. By using temporal information, the vehicle's environmental perception model can be more confident in its predictions of the environment and can also infer the location of temporarily occluded objects. Typically, when utilizing temporal information, we need to use the vehicle's real-world positioning information, namely the vehicle's position coordinates and orientation angle. With positioning information available, multiple bird's-eye view feature maps can be aligned point-by-point.

[0003] In related technologies, the method used for multi-frame semantic fusion temporal information of bird's-eye view feature maps can be a three-dimensional convolutional neural network (3D CNN). Typically, two or three historical bird's-eye view feature maps are directly stitched together in the feature dimension and then input into the 3D CNN.

[0004] Based on the 3D CNN method described above, when fusing the temporal information of the semantics of multiple frames of bird's-eye view feature maps, the processing requires inputting 2 or 3 frames of historical bird's-eye view feature maps simultaneously, resulting in a long latency and low accuracy of the obtained prediction results of the surrounding environment of autonomous vehicles. Summary of the Invention

[0005] This application provides an image processing method and apparatus for reducing the latency required to process multi-frame bird's-eye view feature maps and improving the accuracy of prediction results of the surrounding environment of autonomous vehicles.

[0006] In a first aspect, embodiments of this application provide an image processing method applied to autonomous vehicles, comprising:

[0007] Obtain bird's-eye view feature maps and corresponding localization information of the most recently generated consecutive frames of images. This localization information is the localization information of the autonomous vehicle when the corresponding images were generated. The localization information includes position coordinates and orientation angle.

[0008] For each frame of the multi-frame bird's-eye view feature map, perform the following operations:

[0009] Based on the orientation angle of the autonomous vehicle corresponding to the bird's-eye view feature map, the bird's-eye view feature map is rotated to obtain the rotated bird's-eye view feature map.

[0010] Based on the position coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map, determine the target area occupied by the rotated bird's-eye view feature map in the fused feature map;

[0011] Based on the target region, the rotated bird's-eye view feature map is fused into the fused feature map.

[0012] One possible implementation involves fusing the rotated bird's-eye view feature map into a fused feature map based on the target region, including:

[0013] From the fused feature map, the feature map of the target region is cropped out;

[0014] The rotated bird's-eye view feature map and the feature map of the target area are fused together to obtain the updated bird's-eye view feature map.

[0015] In the fused feature map, the feature map of the target region is replaced with the updated bird's-eye view feature map.

[0016] One possible implementation involves fusing the rotated bird's-eye view feature map and the feature map of the target region to obtain an updated bird's-eye view feature map, including:

[0017] The rotated bird's-eye view feature map and the feature map of the target region are input into the first model for fusion processing to obtain the updated bird's-eye view feature map. The first model includes convGRU.

[0018] In one possible implementation, the target region occupied by the rotated bird's-eye view feature map in the fused feature map is determined based on the position coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map, including:

[0019] Based on the mapping relationship between the location coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map and the fused feature map, the fused coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map in the fused feature map are obtained.

[0020] Based on the rotated bird's-eye view feature map and the fused coordinates, the target area occupied by the rotated bird's-eye view feature map in the fused feature map is obtained.

[0021] One possible implementation involves obtaining the location information corresponding to the image, including:

[0022] The location information of the autonomous vehicle corresponding to the image is obtained from the navigation device of the autonomous vehicle.

[0023] In one possible implementation, after fusing the rotated bird's-eye view feature map into the fused feature map based on the target region, the following steps are also included:

[0024] From the fused feature map, the feature map of the target region corresponding to the bird's-eye view feature map of the corresponding frame is cropped out for use in downstream tasks.

[0025] Secondly, embodiments of this application provide an image processing apparatus for use in autonomous vehicles, the image processing apparatus comprising:

[0026] The acquisition module is used to acquire the bird's-eye view feature map and the corresponding positioning information of the most recently generated consecutive frames of images. The positioning information is the positioning information of the autonomous vehicle when the corresponding image is generated. The positioning information includes position coordinates and orientation angle.

[0027] A processing apparatus for each frame of a multi-frame bird's-eye view feature map, comprising:

[0028] The rotation module is used to rotate the bird's-eye view feature map based on the orientation angle of the autonomous vehicle corresponding to the bird's-eye view feature map, so as to obtain the rotated bird's-eye view feature map.

[0029] The determination module is used to determine the target area occupied by the rotated bird's-eye view feature map in the fused feature map based on the position coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map.

[0030] The fusion module is used to fuse the rotated bird's-eye view feature map into the fused feature map based on the target region.

[0031] In one possible implementation, the fusion module is specifically used for:

[0032] From the fused feature map, the feature map of the target region is cropped out;

[0033] The rotated bird's-eye view feature map and the feature map of the target area are fused together to obtain the updated bird's-eye view feature map.

[0034] In the fused feature map, the feature map of the target region is replaced with the updated bird's-eye view feature map.

[0035] In one possible implementation, the fusion module is also used for:

[0036] The rotated bird's-eye view feature map and the feature map of the target region are input into the first model for fusion processing to obtain the updated bird's-eye view feature map. The first model includes convGRU.

[0037] In one possible implementation, the determination module is specifically used for:

[0038] Based on the mapping relationship between the location coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map and the fused feature map, the fused coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map in the fused feature map are obtained.

[0039] Based on the rotated bird's-eye view feature map and the fused coordinates, the target area occupied by the rotated bird's-eye view feature map in the fused feature map is obtained.

[0040] In one possible implementation, the acquisition module is also used for:

[0041] The location information of the autonomous vehicle corresponding to the image is obtained from the navigation device of the autonomous vehicle.

[0042] One possible implementation also includes a trimming module, which is used for:

[0043] From the fused feature map, the feature map of the target region corresponding to the bird's-eye view feature map of the corresponding frame is cropped out for use in downstream tasks.

[0044] Thirdly, this application provides an image processing apparatus, comprising:

[0045] At least one processor;

[0046] and memory connected to at least one processor;

[0047] The memory is used to store at least one processor-executable instruction, which is executed by at least one processor to enable at least one processor to perform the image processing method provided in the first aspect.

[0048] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the image processing method provided in the first aspect.

[0049] Fifthly, this application provides a program product comprising computer-executable instructions. When the computer-executable instructions are executed, they implement the image processing method provided in the first aspect.

[0050] The image processing method and apparatus provided in this application process each frame of the bird's-eye view feature map of the most recently generated consecutive frames of images separately, based on the orientation angle of the autonomous vehicle corresponding to each frame of the bird's-eye view feature map, to obtain a fused feature map containing multiple frames of bird's-eye view feature maps. This reduces the latency required for processing the bird's-eye view feature maps of consecutive frames of images, thereby improving the accuracy of the prediction results of the surrounding environment of the autonomous vehicle. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0052] Figure 1a A schematic diagram of a bird's-eye view feature map provided for an embodiment of this application;

[0053] Figure 1b A schematic diagram of a fused bird's-eye view feature map of one historical frame and three historical frames provided for embodiments of this application;

[0054] Figure 2 A flowchart of the image processing method provided in Embodiment 1 of this application;

[0055] Figure 3 A schematic diagram illustrating the process of rotating any given frame of bird's-eye view according to the orientation angle of the corresponding autonomous vehicle.

[0056] Figure 4 A flowchart of the image processing method provided in Embodiment 2 of this application;

[0057] Figure 5 A schematic diagram illustrating the process of fusing eight consecutive historical bird's-eye view feature maps.

[0058] Figure 6 A flowchart of the image processing method provided in Embodiment 3 of this application;

[0059] Figure 7 A schematic diagram of the target region occupied by the rotated bird's-eye view feature map 32 in the fused feature map;

[0060] Figure 8 This is a schematic diagram of the image processing apparatus provided in Embodiment 4 of this application;

[0061] Figure 9 This is a schematic diagram of the image processing device provided in Embodiment 5 of this application.

[0062] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0064] In autonomous driving, utilizing the temporal information contained in multi-frame bird's-eye view feature maps is particularly important. By leveraging this temporal information, firstly, the environmental perception model can be more confident in its predictions because it has seen the current environment in the previous frame; secondly, based on information from consecutive frames of bird's-eye view feature maps, the environmental perception model can also infer the location of temporarily occluded objects. Typically, when utilizing temporal information, we need the vehicle's real-world positioning information, including its position coordinates and orientation angle. With positioning information available, multiple bird's-eye view feature maps can be aligned point-by-point.

[0065] Currently, there are two main methods for fusing temporal information from multiple frames of bird's-eye view feature maps: 3D CNN and Transformer models. The 3D CNN method requires concatenating two or three historical bird's-eye view feature maps along their feature dimensions before inputting them into the 3D CNN. The Transformer model method typically requires interaction between each feature point on the two or three historical bird's-eye view feature maps. Both methods require simultaneous input of two or three historical bird's-eye view feature maps for processing, resulting in high processing costs, long latency, and significant GPU memory consumption. Furthermore, these two methods can only fuse temporal information from two or three historical bird's-eye view feature maps, leading to relatively low accuracy in predicting the surrounding environment of autonomous vehicles.

[0066] To address the aforementioned issues, this application embodiment constructs a blank feature map. Each frame from multiple consecutive historical bird's-eye view feature maps is sequentially input into the blank feature map in chronological order, and the blank feature map is updated sequentially to obtain a fused feature map containing chronological information from multiple historical bird's-eye view feature maps. This reduces processing costs, shortens processing latency, and decreases GPU memory usage during processing. Furthermore, it improves the accuracy of environmental prediction results for autonomous vehicles.

[0067] To facilitate understanding, the bird's-eye view and the fusion of bird's-eye feature maps in the embodiments of this application will be introduced first.

[0068] Figure 1a This is a schematic diagram of a bird's-eye view feature map provided for an embodiment of this application. For example... Figure 1a The images shown are four bird's-eye view feature maps under different environments. Each bird's-eye view feature map includes moving vehicles and lanes. Vehicles are indicated by small rectangles in the images, with the vehicles within the dashed boxes representing autonomous vehicles. Figure 1a The four bird's-eye view feature maps show that the bird's-eye view can clearly show the environment around autonomous vehicles.

[0069] Currently, in the field of autonomous driving, bird's-eye view feature maps are mainly obtained through LiDAR models and pure vision models. Specifically, the LiDAR model acquires point clouds in 3D space and transforms the corresponding point cloud maps into bird's-eye view features; the pure vision model uses six cameras installed on the autonomous vehicle in different directions to capture images of the autonomous vehicle and its surrounding environment, and converts the captured images into bird's-eye view features. The bird's-eye view feature maps obtained through either method are input into the autonomous vehicle's environmental perception model. The environmental perception model fuses two or three historical frames of bird's-eye view feature maps based on temporal information to obtain a bird's-eye view feature map of the autonomous vehicle's surrounding environment within a certain timeframe. Based on the obtained bird's-eye view feature maps, the driving safety of autonomous vehicles can be ensured.

[0070] Figure 1b This is a schematic diagram of a fused bird's-eye view feature map derived from a historical 1-frame bird's-eye view feature map and three historical 3-frame bird's-eye view feature maps, as provided in an embodiment of this application. Figure 1b As shown, (1) is the historical 1-frame bird's-eye view feature map, and (2) is the fused feature map of the historical 3-frame bird's-eye view feature map. Specifically, the historical 3-frame bird's-eye view feature map consists of 3 consecutive frames, and the temporal order of the 3 frames is shown in the numerical order in the figure. Among them, the dark gray area is the feature of the lane, the light gray area is the environmental feature around the lane, and the dashed line is the lower boundary of each frame bird's-eye view feature map.

[0071] In autonomous driving, utilizing the temporal information contained in multi-frame bird's-eye view feature maps is particularly important. Typically, utilizing temporal information requires obtaining the autonomous vehicle's location information, i.e., its real-time position coordinates and orientation angle. With the autonomous vehicle's location information, feature point-by-feature alignment of multiple bird's-eye view feature maps can be achieved. For example... Figure 1b As shown, the features displayed are that as the vehicle moves forward, the alignment and stitching of feature points are achieved based on the bird's-eye view feature map and its corresponding positioning information, resulting in a bird's-eye view feature map after the fusion of multiple bird's-eye view feature maps.

[0072] The following is combined Figure 1a Bird's-eye view features and Figure 1b The image processing method provided in this application embodiment is described in detail by fusing three historical bird's-eye view feature maps.

[0073] Figure 2 This is a flowchart of the image processing method provided in Embodiment 1 of this application. Figure 2 As shown, the image processing method includes the following steps:

[0074] S201, Obtain the bird's-eye view feature maps and corresponding localization information of the most recently generated consecutive frames of images.

[0075] During the operation of an autonomous vehicle, the environmental perception model used to output a bird's-eye view feature map of the autonomous vehicle and its surrounding environment is updated periodically. For example, the update cycle can be 0.1 seconds. Therefore, the bird's-eye view feature maps of the autonomous vehicle and its surrounding environment output by the environmental perception model are all recently generated in real time.

[0076] Bird's-eye view feature maps of consecutive multi-frame images can be obtained through different methods. One possible implementation is through a LiDAR model installed on an autonomous vehicle. Specifically, the LiDAR model acquires point clouds in three-dimensional space and transforms the corresponding point cloud maps into features from a bird's-eye view perspective, i.e., consecutive multi-frame bird's-eye view feature maps with temporal information.

[0077] Another possible implementation is through a pure vision model installed on an autonomous vehicle. Specifically, the pure vision model uses six cameras installed on the autonomous vehicle in different directions to capture images of the vehicle and its surrounding environment, and converts the captured images into features with a bird's-eye view perspective, i.e., a continuous multi-frame bird's-eye view feature map with temporal information.

[0078] The positioning information corresponding to each frame of the image is the positioning information of the autonomous vehicle at the time the corresponding image was generated. This positioning information includes position coordinates and orientation angle, that is, the real-time position coordinates and orientation angle of the autonomous vehicle in the real world. For example, the positioning information of the autonomous vehicle can be the position coordinates of the vehicle in the Universal Transverse Mercator Grid System (UTM) coordinate system and the orientation angle of the vehicle relative to due east. In one possible implementation, the positioning information corresponding to the autonomous vehicle in each frame of the image can be obtained from the navigation device of the autonomous vehicle.

[0079] S202, For each frame of the multi-frame bird's-eye view feature map, perform the following operations.

[0080] In some embodiments, this step may specifically include steps S2021 to S2023:

[0081] S2021, based on the orientation angle of the autonomous vehicle corresponding to the bird's-eye view feature map, rotate the bird's-eye view feature map to obtain the rotated bird's-eye view feature map.

[0082] Optionally, the dimensions of each frame's bird's-eye view feature map can be C×H×W. Here, C represents the feature dimension, and H and W represent the height and width of the bird's-eye view feature map, respectively.

[0083] Optionally, the location information of the autonomous vehicle corresponding to multiple consecutive bird's-eye view frames can be represented by a set L. Specifically, it can be represented as L = {L1, L2, ..., L...} N}. For the location information L of the autonomous vehicle corresponding to the bird's-eye view feature map of the i-th frame. i This can be expressed as: orientation angle α i and position coordinates (x) i y i ).

[0084] In one possible implementation, the heading angle is defined as negative when the autonomous vehicle deviates to the left relative to due east, and positive when it deviates to the right relative to due east. For example, when the autonomous vehicle deviates 5° to the left relative to due east, the heading angle can be represented as α. i = -5°; When an autonomous vehicle deviates 5° to the right relative to due east, the heading angle can be expressed as α. i =5°. Each frame of the bird's-eye view is rotated according to the orientation angle of the autonomous driving system corresponding to each frame. When the orientation angle is negative, it rotates counterclockwise; when the orientation angle is positive, it rotates clockwise.

[0085] Figure 3 This is a schematic diagram illustrating the process of rotating any given frame of a bird's-eye view according to the orientation angle of the corresponding autonomous vehicle. (Example:) Figure 3 As shown, the height and width of the bird's-eye view feature map 31 are H and W, respectively, and its corresponding autonomous driving orientation angle is 30°. The height and width of the rotated bird's-eye view feature map 32 are H1 and W1, respectively.

[0086] For example, the height and width of the bird's-eye view feature image 31 and the rotated bird's-eye view feature image 32 can be the same or different. This application embodiment does not limit whether the height and width of the bird's-eye view feature images change before and after rotation.

[0087] By rotating the bird's-eye view feature map according to the orientation angle, the driving trajectory of autonomous vehicles can be made consistent with the driving trajectory on actual roads, further ensuring the accuracy of the prediction of the surrounding environment of autonomous vehicles during driving.

[0088] S2022, Based on the position coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map, determine the target area occupied by the rotated bird's-eye view feature map in the fused feature map.

[0089] The fused feature map initially starts as a blank image without any features and is generated within the environmental perception model of the autonomous vehicle. For example, the dimensions of the fused feature map can be C′×H′×W′, where C′ represents the feature dimension, and H′ and W′ represent the height and width of the fused feature map, respectively. In one possible implementation, the height and width of the fused feature map are much larger than those of the bird's-eye view feature map.

[0090] Based on the location coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map, the position of the autonomous vehicle in the fused feature map is determined, and the target area that the rotated bird's-eye view feature map needs to occupy in the fused feature map is delineated according to the height and width of the rotated bird's-eye view feature map.

[0091] S2023, based on the target region, merges the rotated bird's-eye view feature map into the fused feature map.

[0092] The target region is part of the fused feature map. By fusing the rotated bird's-eye view feature map into the target region, the target region acquires a bird's-eye view feature map, which can then be used to display the surrounding environment of the autonomous vehicle during its operation.

[0093] For the rotation and fusion of multiple consecutive bird's-eye view feature maps, one possible implementation involves performing steps S2021 to S2023 on each bird's-eye view feature map in a temporal sequence. That is, one bird's-eye view feature map is fused into the fused feature map in a temporal sequence, thereby achieving the fusion of multiple consecutive bird's-eye view feature maps.

[0094] In another possible implementation, each frame of the bird's-eye view feature map is processed in step S2021 in time sequence. Then, each frame of the bird's-eye view map processed in step S2021 is processed in time sequence in step S2022. Finally, each frame of the bird's-eye view map processed in step S2022 is fused into the fused feature map in time sequence, thereby realizing the fusion of multiple consecutive frames of bird's-eye view feature maps.

[0095] It is understood that the image processing method provided in this application involves processing the images sequentially and then fusing the processed multi-frame bird's-eye view feature maps into a fused feature map in sequence, resulting in a fused feature map containing multiple bird's-eye view feature maps. Compared to methods in related technologies, this application processes each frame separately and fuses multiple bird's-eye view feature maps into a single fused feature map that does not contain any feature information. Therefore, the method provided in this application can achieve the fusion of historical multi-frame bird's-eye view feature maps. For example, it can be 10 frames, 20 frames, etc.

[0096] In this embodiment, the method acquires bird's-eye view feature maps and corresponding positioning information from recently generated consecutive frames of images. For each frame, the bird's-eye view feature map is rotated based on the orientation angle of the autonomous vehicle corresponding to it, resulting in a rotated bird's-eye view feature map. The target region occupied by the rotated bird's-eye view feature map in the fused feature map is determined based on the position coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map. Based on the target region, the rotated bird's-eye view feature map is fused into the fused feature map. This method, by processing each frame of bird's-eye view image separately, can shorten the processing latency, reduce processing costs, and decrease the GPU memory usage. Furthermore, by sequentially fusing multiple frames of bird's-eye view feature maps into the fused feature map, the fusion of historical multiple frames of bird's-eye view feature maps can be achieved, thereby improving the utilization efficiency of feature information from multiple frames of bird's-eye view images and the accuracy of the prediction results for the surrounding environment of the autonomous vehicle.

[0097] Figure 4 This is a flowchart of the image processing method provided in Embodiment 2 of this application. This embodiment is a detailed description of step S2023 in Embodiment 1. Figure 4 As shown, step S2023 may further include the following steps:

[0098] S401, crop the feature map of the target region from the fused feature map.

[0099] Based on the position and size of the target region in the fused feature map, the target region is cropped out from the fused feature map to obtain the feature map of the target region.

[0100] S402, the rotated bird's-eye view feature map and the feature map of the target area are fused to obtain the updated bird's-eye view feature map.

[0101] Optionally, the rotated bird's-eye view feature map and the feature map of the target region are input into a first model for fusion processing to obtain an updated bird's-eye view feature map. For example, the first model includes a convGRU neural network model. Specifically, the feature dimension C′ of the fused feature map can be the same as the dimension of the convGRU neural network model. In one possible implementation, the feature dimension C of each frame's bird's-eye view feature map is smaller than the feature dimension C′ of the fused feature map; for example, the feature dimension C′ of the fused feature map can be twice the feature dimension C of each frame's bird's-eye view feature map.

[0102] The height and width of the updated bird's-eye view feature map are the same as the height and width of the target area, i.e., the rotated bird's-eye view, and it has the characteristics of an autonomous vehicle bird's-eye view.

[0103] S403, in the fused feature map, replace the feature map of the target region with the updated bird's-eye view feature map.

[0104] The updated bird's-eye view feature map is placed at the position of the target region feature map in the fused feature map, that is, the bird's-eye view feature map is fused into the fused feature map. Optionally, after fusing all historical bird's-eye view feature maps from multiple consecutive frames into the fused feature map in chronological order, the feature map of the target region corresponding to the bird's-eye view feature map of the corresponding frame is cropped from the fused feature map for use in downstream tasks. Specifically, the feature map of the target region corresponding to the bird's-eye view feature map of the corresponding frame contains the target region features of all bird's-eye view feature maps fused before the corresponding frame.

[0105] Figure 5 This is a schematic diagram illustrating the process of fusing eight consecutive historical aerial view feature maps. (Example) Figure 5 As shown, 5a is a schematic diagram of fusing the first frame of bird's-eye view feature map into the fused feature map, 5b is a schematic diagram of fusing the second frame of bird's-eye view feature map into the fused feature map in sequence, where the top frame is the first frame of bird's-eye view feature map, the bottom frame is the second frame of bird's-eye view feature map, and so on. 5e is a schematic diagram of fusing 6 consecutive frames of bird's-eye view feature maps, and 5f is a schematic diagram of fusing 8 consecutive frames of bird's-eye view feature maps. From the fused feature map, the feature map of the target area corresponding to the 8th frame of bird's-eye view feature map is cropped to obtain the fused map of 8 frames of bird's-eye view feature maps, which is used to reflect the environment around the autonomous vehicle. It is understood that the following processing method provided in the embodiments of this application does not limit the number of frames of bird's-eye view feature maps that can be fused. For example, it can be 3 frames or 20 frames.

[0106] In this embodiment, the feature map of the target region is cropped from the fused feature map; the rotated bird's-eye view feature map and the feature map of the target region are fused to obtain an updated bird's-eye view feature map; in the fused feature map, the feature map of the target region is replaced with the updated bird's-eye view feature map. This method fuses the bird's-eye view feature map into the fused feature map. By temporally fusing multiple consecutive frames of historical bird's-eye view feature patterns into the fused feature map, a fused multi-frame bird's-eye view feature map can be obtained, achieving accurate prediction of the environment surrounding autonomous vehicles.

[0107] Figure 6 This is a flowchart of the image processing method provided in Embodiment 3 of this application. This embodiment is a detailed description of step S2022 in Embodiment 1. Figure 6 As shown, step S2022 may further include the following steps:

[0108] S601, based on the mapping relationship between the position coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map and the fused feature map, obtain the fused coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map in the fused feature map.

[0109] The mapping relationship can be the correspondence between the location coordinates of the autonomous vehicle in the bird's-eye view feature map and the location coordinates of each pixel in the fused feature map.

[0110] In one possible implementation, the top-left corner of the fused feature map is defined as the origin coordinate. The average coordinates of the autonomous vehicle's position coordinates corresponding to the locations in multiple consecutive historical bird's-eye view feature maps are calculated. These average coordinates are then mapped to the coordinates of the center pixel in the fused feature map. Based on this mapping, the pixel coordinates in the fused feature map corresponding to the autonomous vehicle's position coordinates in each specific bird's-eye view feature map are obtained. In other words, these are the fused coordinates of the autonomous vehicle in the fused feature map corresponding to the bird's-eye view feature map.

[0111] S602, based on the rotated bird's-eye view feature map and the fused coordinates, obtain the target area occupied by the rotated bird's-eye view feature map in the fused feature map.

[0112] For example, Figure 7 This is a schematic diagram of the target region occupied by the rotated bird's-eye view feature map 32 in the fused feature map. For example... Figure 7 As shown, the height and width of the fused feature map 71 are H′ and W′, respectively. The height and width of the target region 72 are the same as those of the rotated bird's-eye view feature map 32, which are H1 and W1, respectively. The fused coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map in the fused feature map are (X... i Y i For example, the fused coordinates can be located at the center of the target region. For example, when the top-left corner of the fused feature map is the origin, the coordinates of the top-left and bottom-right corners of the target region can be obtained as (X...). i -W1 / 2, Y i -H1 / 2) and (X i +W1 / 2, Y i +H1 / 2).

[0113] In this embodiment, by mapping the position coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map to the fused feature map, the fused coordinates of the autonomous vehicle in the fused feature map are obtained. Based on the rotated bird's-eye view feature map and the fused coordinates, the target area occupied by the rotated bird's-eye view feature map in the fused feature map is obtained. This method ensures the true position of the autonomous vehicle in the bird's-eye view feature map by rotating it, and completely integrates the rotated bird's-eye view feature map into the fused feature map, thereby improving the accuracy of predicting the surrounding environment during the autonomous vehicle's operation.

[0114] The following are embodiments of the apparatus of this application, which can be used to execute the embodiments of the method of this application. For details not disclosed in the embodiments of the apparatus of this application, please refer to the embodiments of the method of this application.

[0115] Figure 8This is a schematic diagram of the image processing apparatus provided in Embodiment 4 of this application. The image processing apparatus provided in this embodiment is applied to an autonomous vehicle. Figure 8 As shown, the image processing device 80 includes: an acquisition module 810, a rotation module 820, a determination module 830, and a fusion module 840.

[0116] The acquisition module 810 is used to acquire the bird's-eye view feature map of the most recently generated consecutive multi-frame images and the corresponding positioning information. The positioning information is the positioning information of the autonomous vehicle when the corresponding image is generated. The positioning information includes position coordinates and orientation angle.

[0117] A processing apparatus for each frame of a multi-frame bird's-eye view feature map, comprising:

[0118] The rotation module 820 is used to rotate the bird's-eye view feature map based on the orientation angle of the autonomous vehicle corresponding to the bird's-eye view feature map, so as to obtain the rotated bird's-eye view feature map.

[0119] The determination module 830 is used to determine the target area occupied by the rotated bird's-eye view feature map in the fused feature map based on the position coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map.

[0120] The fusion module 840 is used to fuse the rotated bird's-eye view feature map into the fused feature map based on the target region.

[0121] In one possible implementation, the fusion module 840 is specifically used for:

[0122] From the fused feature map, the feature map of the target region is cropped out;

[0123] The rotated bird's-eye view feature map and the feature map of the target area are fused together to obtain the updated bird's-eye view feature map.

[0124] In the fused feature map, the feature map of the target region is replaced with the updated bird's-eye view feature map.

[0125] In one possible implementation, the fusion module 840 is also used for:

[0126] The rotated bird's-eye view feature map and the feature map of the target region are input into the first model for fusion processing to obtain the updated bird's-eye view feature map. The first model includes convGRU.

[0127] In one possible implementation, the determining module 830 is specifically used for:

[0128] Based on the mapping relationship between the location coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map and the fused feature map, the fused coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map in the fused feature map are obtained.

[0129] Based on the rotated bird's-eye view feature map and the fused coordinates, the target area occupied by the rotated bird's-eye view feature map in the fused feature map is obtained.

[0130] In one possible implementation, the acquisition module 810 is also used for:

[0131] The location information of the autonomous vehicle corresponding to the image is obtained from the navigation device of the autonomous vehicle.

[0132] One possible implementation also includes a trimming module, which is used for:

[0133] From the fused feature map, the feature map of the target region corresponding to the bird's-eye view feature map of the corresponding frame is cropped out for use in downstream tasks.

[0134] The image processing apparatus provided in this embodiment can be used to execute the method steps of the above method embodiments. The specific implementation and technical effects are similar, and will not be described again here.

[0135] Figure 9 This is a schematic diagram of the image processing device provided in Embodiment 5 of this application. Figure 9 As shown, the image processing device 90 includes:

[0136] At least one processor 901; and

[0137] A memory 902 is communicatively connected to at least one processor 901; wherein,

[0138] The memory 902 stores instructions that can be executed by at least one processor 901, which enables the at least one processor 901 to perform the method steps in the above-described method embodiments.

[0139] The specific implementation process of processor 901 can be found in the above method embodiments. The specific implementation method and technical effect are similar, and will not be repeated here.

[0140] Optionally, the image processing device may be specifically an autonomous vehicle or a drone, or the electronic device may be integrated into an autonomous vehicle, drone, or other similar device.

[0141] This application also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, the computer-executable instructions are used to implement the method steps as described in the above method embodiments. The specific implementation methods and technical effects are similar and will not be repeated here.

[0142] This application also provides a program product comprising computer-executable instructions. When the computer-executable instructions are executed, they implement the method steps as described in the above method embodiments. The specific implementation and technical effects are similar and will not be repeated here.

[0143] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0144] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The image processing method, applied to autonomous vehicles, includes: Obtain bird's-eye view feature maps and corresponding positioning information of the most recently generated consecutive multi-frame images. The positioning information is the positioning information of the autonomous vehicle when the corresponding images are generated, and the positioning information includes position coordinates and orientation angle. Obtaining the location information corresponding to the image includes: obtaining the location information of the autonomous vehicle corresponding to the image from the navigation device of the autonomous vehicle; For each frame of the multi-frame bird's-eye view feature map, perform the following operations: Based on the orientation angle of the autonomous vehicle corresponding to the bird's-eye view feature map, the bird's-eye view feature map is rotated to obtain a rotated bird's-eye view feature map. Based on the position coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map, determine the target area occupied by the rotated bird's-eye view feature map in the fused feature map; Based on the target region, the rotated bird's-eye view feature map is fused into the fused feature map; The fused feature map is initially a blank image without any features, generated in the environmental perception model of the autonomous vehicle, and the height and width of the fused feature map are much larger than the height and width of the bird's-eye view feature map. By rotating and fusing each frame of the bird's-eye view feature map in time sequence, multiple consecutive frames of bird's-eye view feature maps are fused into the fused feature map to obtain a fused feature map with time sequence information of multiple frames of historical bird's-eye view feature maps.

2. The method according to claim 1, characterized in that, The step of fusing the rotated bird's-eye view feature map into the fused feature map based on the target region includes: From the fused feature map, the feature map of the target region is cropped; The rotated bird's-eye view feature map and the feature map of the target area are fused together to obtain an updated bird's-eye view feature map. In the fused feature map, the feature map of the target region is replaced with the updated bird's-eye view feature map.

3. The method according to claim 2, characterized in that, The step of fusing the rotated bird's-eye view feature map and the feature map of the target region to obtain an updated bird's-eye view feature map includes: The rotated bird's-eye view feature map and the feature map of the target region are input into a first model for fusion processing to obtain an updated bird's-eye view feature map. The first model includes convGRU.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the target region occupied by the rotated bird's-eye view feature map in the fused feature map based on the position coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map includes: Based on the mapping relationship between the location coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map and the fused feature map, the fused coordinates of the autonomous vehicle corresponding to the bird's-eye view feature map in the fused feature map are obtained; Based on the rotated bird's-eye view feature map and the fused coordinates, the target area occupied by the rotated bird's-eye view feature map in the fused feature map is obtained.

5. The method according to any one of claims 1 to 3, characterized in that, After fusing the rotated bird's-eye view feature map into the fused feature map based on the target region, the method further includes: From the fused feature map, the feature map of the target region corresponding to the bird's-eye view feature map of the corresponding frame is cropped out for use in downstream tasks.

6. An image processing apparatus, characterized in that, The image processing device, used in autonomous vehicles, includes: The acquisition module is used to acquire bird's-eye view feature maps and corresponding positioning information of the most recently generated consecutive multi-frame images. The positioning information is the positioning information of the autonomous vehicle when the corresponding image is generated, and the positioning information includes position coordinates and orientation angle. The acquisition module is specifically used to acquire the location information of the autonomous vehicle corresponding to the image from the navigation device of the autonomous vehicle; A processing apparatus for each frame of a multi-frame bird's-eye view feature map, comprising: A rotation module is used to rotate the bird's-eye view feature map based on the orientation angle corresponding to the autonomous vehicle in the bird's-eye view feature map, so as to obtain a rotated bird's-eye view feature map. The determination module is used to determine the target area occupied by the rotated bird's-eye view feature map in the fused feature map based on the position coordinates of the autonomous vehicle in the bird's-eye view feature map. The fusion module is used to fuse the rotated bird's-eye view feature map into a fused feature map based on the target region. The fused feature map is initially a blank image without any features, generated in the environmental perception model of the autonomous vehicle, and the height and width of the fused feature map are much larger than the height and width of the bird's-eye view feature map. By rotating and fusing each frame of the bird's-eye view feature map in time sequence, multiple consecutive frames of bird's-eye view feature maps are fused into the fused feature map to obtain a fused feature map with time sequence information of multiple frames of historical bird's-eye view feature maps.

7. The apparatus according to claim 6, characterized in that, The fusion module is specifically used for: From the fused feature map, the feature map of the target region is cropped; The rotated bird's-eye view feature map and the feature map of the target area are fused together to obtain an updated bird's-eye view feature map. In the fused feature map, the feature map of the target region is replaced with the updated bird's-eye view feature map.

8. An image processing device, characterized in that, include: At least one processor; and a memory connected to the at least one processor; The memory is used to store instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, are used to implement the method as described in any one of claims 1 to 5.

Images