Motion information estimation method, apparatus, device, storage medium and program product

Abstract

Embodiments of the present disclosure relate to a motion information estimation method, device, equipment, storage medium and program product. The method comprises: obtaining a pseudo-image feature map corresponding to original point cloud data; inputting the pseudo-image feature map into a preset motion estimation network for motion estimation processing to determine a target feature map corresponding to the pseudo-image feature map; determining a scene flow corresponding to the original point cloud data according to the target feature map; the scene flow comprising motion information of each point in an environment corresponding to the original point cloud data; wherein the motion estimation network comprises a plurality of motion estimation layers, and the estimation result of any one motion estimation layer in the plurality of motion estimation layers is used as the input of the adjacent next motion estimation layer. The method can ensure the accuracy of the estimated motion information.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of vehicle automatic driving, and in particular, to a motion information estimation method, device, equipment, storage medium and program product. BACKGROUND

[0002] An automatic driving vehicle refers to a vehicle in which various sensors are installed, the surrounding environment information is perceived through the various sensors, and then the motion information, semantic information, position information, etc. of the surrounding environment are obtained through processing of the data perceived by the various sensors, wherein the estimation of the motion information can help the automatic driving vehicle to better perceive dynamic obstacles and calculate the motion trajectory of an object, and finally be used for path planning of the vehicle, etc. to assist the vehicle in automatic driving.

[0003] However, the current technology cannot guarantee the accuracy of the estimated motion information. SUMMARY

[0004] Embodiments of the present disclosure provide a motion information estimation method, device, equipment, storage medium and program product, which can guarantee the accuracy of the estimated motion information.

[0005] In a first aspect, embodiments of the present disclosure provide a motion information estimation method, which comprises:

[0006] obtaining a pseudo-image feature map corresponding to original point cloud data;

[0007] inputting the pseudo-image feature map into a preset motion estimation network for motion estimation processing to determine a target feature map corresponding to the pseudo-image feature map; wherein the motion estimation network comprises multiple motion estimation layers, and the estimation result of any one motion estimation layer in the multiple motion estimation layers is used as the input of the adjacent next motion estimation layer.

[0008] determining a scene flow corresponding to the original point cloud data according to the target feature map; wherein the scene flow comprises the motion information of each point in the environment corresponding to the original point cloud data.

[0009] In a second aspect, embodiments of the present disclosure provide a motion information estimation device, which comprises:

[0010] an acquisition module configured to obtain a pseudo-image feature map corresponding to original point cloud data;

[0011] a motion estimation module configured to input the pseudo-image feature map into a preset motion estimation network for motion estimation processing to determine a target feature map corresponding to the pseudo-image feature map; wherein the motion estimation network comprises multiple motion estimation layers, and the estimation result of any one motion estimation layer in the multiple motion estimation layers is used as the input of the adjacent next motion estimation layer.

[0012] determining module configured to determine a scene flow corresponding to the original point cloud data according to the target feature map; the scene flow includes motion information of each point in an environment corresponding to the original point cloud data.

[0013] In a third aspect, an embodiment of the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method in the first aspect when executing the computer program.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the method in the first aspect.

[0015] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, and the computer program is executable on a processor to implement the method in the first aspect.

[0016] The motion information estimation method, device, equipment, storage medium and program product provided by the embodiments of the present disclosure can obtain a pseudo-image feature map corresponding to original point cloud data, input the pseudo-image feature map into a preset motion estimation network for motion estimation processing, determine a target feature map corresponding to the pseudo-image feature map, and then determine a scene flow corresponding to the original point cloud data according to the target feature map; the scene flow includes motion information of each point in an environment corresponding to the original point cloud data, and the motion estimation network includes multiple motion estimation layers, and the estimation result of any motion estimation layer in the multiple motion estimation layers is used as the input of the adjacent next motion estimation layer. In the method, when the motion information in the scene flow is estimated by the motion estimation network including the multiple motion estimation layers, the estimation result of any motion estimation layer can be used as the input of the adjacent next motion estimation layer, so that the motion estimation of the next layer can be assisted by the motion estimation result of the previous layer, and the information combined during estimation is more abundant, so that the target feature map estimated is more accurate, and the motion information estimated through the target feature map is more accurate, thereby ensuring the accuracy of the finally estimated motion information. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 FIG. 1 is a network architecture diagram of motion information estimation in related technology 1;

[0018] Figure 2 FIG. 2 is a network architecture diagram of motion information estimation in related technology 2;

[0019] Figure 3 FIG. 3 is an internal structure diagram of a computer device in an embodiment;

[0020] Figure 4This is a flowchart illustrating a motion information estimation method in one embodiment;

[0021] Figure 5 This is a flowchart illustrating the motion information estimation method in another embodiment;

[0022] Figure 6 This is a flowchart illustrating the motion information estimation method in another embodiment;

[0023] Figure 7 Here is a block diagram of the motion estimation network architecture in another embodiment;

[0024] Figure 8 This is a flowchart illustrating the motion information estimation method in another embodiment;

[0025] Figure 9 Here is an overall flowchart of the motion information estimation method in another embodiment;

[0026] Figure 10 This is a structural block diagram of a motion information estimation device in one embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this disclosure and are not intended to limit the embodiments of this disclosure.

[0028] First, before introducing the technical solutions of the embodiments of this disclosure in detail, the technical background or evolution of the embodiments of this disclosure will be introduced. Generally, in the field of autonomous driving, the current technical background is that it is generally necessary to estimate the motion information of autonomous vehicles in order to better assist autonomous driving. Currently, there are two estimation methods for motion information: one is to use... Figure 1 Based on the PWC net shown (or similar algorithms like RAFT), motion information estimation is performed. This approach first uses an upper-level pyramid to estimate larger-amplitude motions, then upsamples the estimated optical flow. This optical flow is then used to offset and distort features, serving as input for the next layer's optical flow estimation. The next layer estimates the correction amount for the previous layer's optical flow estimation result. In this approach, the amplitude of dense optical flow varies at each location in the image. The upper-level pyramid has a larger receptive field, enabling the computation of larger-amplitude optical flow values. Furthermore, as the optical flow is passed down from top to bottom, the accuracy of the optical flow can be progressively improved, resulting in a more refined and accurate estimation. However, this approach involves a relatively large computational load in the network structure, leading to slower motion estimation. Another approach is to use... Figure 2The Motion Net shown performs motion information estimation. In the Motion Net, a space-time convolution module (STC Block) can be used to perform time and space down-sampling convolution operations on the input data to obtain convolution features. Then, the corresponding convolution features are deconvoluted and two-dimensional convolution processed, and finally the estimated motion information is obtained. However, for large amplitude motion estimation, the bottom layer pyramid often estimates incorrectly, which affects the optical flow result, that is, the accuracy of the estimated motion information cannot be guaranteed.

[0029] Based on this background, the applicant found through long-term model simulation research and verification that improving the Motion Net motion network can not only guarantee the accuracy of the estimated motion information, but also have a faster motion estimation speed.

[0030] It should be noted that the applicant has made a lot of creative labor for the problems of the above-mentioned several ways of determining the background and the technical solutions introduced in the following embodiments.

[0031] The technical solutions related to the embodiments of the present disclosure will be introduced below in combination with the scenarios to which the embodiments of the present disclosure are applied.

[0032] The motion information estimation method provided by the embodiments of the present disclosure can be applied to a computer device. The computer device can be a terminal, and its internal structure diagram can be as shown in Figure 3 The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless communication can be achieved through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement a motion information estimation method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device. It can also be an external keyboard, touchpad or mouse, etc.

[0033] Those skilled in the art can understand that Figure 3The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0034] In one embodiment, as shown in Figure 4 , a motion information estimation method is provided, which is applied to a computer device as an example in Figure 1 , and can include the following steps:

[0035] S202, obtaining a pseudo-image feature map corresponding to the original point cloud data.

[0036] In this step, the original point cloud data can be obtained by collecting the environment by a sensor. The environment in the original point cloud data can include a vehicle and other objects around the vehicle, etc. The original point cloud data can be three-dimensional point cloud data.

[0037] The sensor can be a laser radar, so that the original point cloud data collected can be laser radar point cloud data. Of course, the sensor can also be other types of sensors, such as a millimeter wave radar. The sensor can be set on the vehicle, or can be set on the roadside, or can be set according to the actual situation.

[0038] After the sensor collects the original point cloud data of the environment, the original point cloud data can be sent to the computer device connected with the sensor, so that the computer device can obtain the original point cloud data and perform subsequent motion estimation processing.

[0039] After the computer device obtains the original point cloud data of the environment, the original point cloud data can be converted into a pseudo-image feature map by using an image conversion related algorithm or a neural network, etc. The pseudo-image feature map here can be a feature map including multiple dimension features, which can represent the features related to the environment on the original point cloud data. As an optional embodiment, the pseudo-image feature map here can include four dimension features, which are time dimension, feature dimension, length of two-dimensional plane and width of two-dimensional plane. The two-dimensional plane here can be the plane projected by the original point cloud data, which can be generally XY plane. The length and width of the two-dimensional plane here can be the length and width of each two-dimensional pseudo-image feature map in the pseudo-image feature map. The time dimension here can be the dimension of the time information of each point in the original point cloud data, and the feature dimension can be the dimension of the feature information related to the environment in the original point cloud data.

[0040] Here, by converting the original point cloud data into a pseudo image feature map, the 2D convolution can be performed by skipping the complex 3D convolution, so that the efficiency of subsequent motion estimation can be improved.

[0041] S204, input the pseudo image feature map into a preset motion estimation network for motion estimation processing to determine a target feature map corresponding to the pseudo image feature map; wherein the motion estimation network includes multiple motion estimation layers, and the estimation result of any motion estimation layer in the multiple motion estimation layers is used as the input of the adjacent next motion estimation layer.

[0042] In this step, the motion estimation network can be a neural network, and the architecture of the motion estimation network can be a pyramid network architecture, which can include multiple motion estimation layers, and each motion estimation layer can include at least one convolution module for convolution processing of the pseudo image feature map to achieve motion estimation and obtain the motion estimation result of the motion estimation layer.

[0043] The multiple motion estimation layers in the motion estimation network are sequentially and adjacently arranged, and from the top motion estimation layer, the estimation result of the previous motion estimation layer can be used as the input of the next motion estimation layer, so that the next motion estimation layer can combine the motion estimation result of the previous motion estimation layer to assist the next motion estimation layer to perform more accurate motion estimation and obtain more accurate motion estimation result.

[0044] Specifically, after obtaining the pseudo image feature map, the pseudo image feature map can be input from the bottom of the motion estimation layer to the motion estimation network, and the pseudo image feature map is processed by each motion estimation layer in the motion estimation network, and the motion estimation result of each motion estimation layer is input to the next motion estimation layer to assist the next motion estimation layer to perform more accurate motion estimation, and finally the motion estimation result is obtained through the motion estimation network.

[0045] The motion estimation result here can be a feature map obtained by fine processing of the pseudo image feature map, denoted as a target feature map, and the accuracy of the target feature map is higher than that of the pseudo image feature map. Meanwhile, the target feature map can be a target feature map after integrating the time information and not including the time information, but the feature dimension in the target feature map is generally a multi-dimensional feature.

[0046] S206, determine the scene flow corresponding to the original point cloud data according to the target feature map; the scene flow includes the motion information of each point in the environment corresponding to the original point cloud data.

[0047] In this step, after obtaining the target feature map with higher precision, the scene flow and the motion information therein can be estimated. As an optional embodiment, the target feature map can be input into a preset motion head estimation network for processing to obtain a scene flow corresponding to the original point cloud data. The scene flow is a two-dimensional scene flow.

[0048] The scene flow refers to a dense scene flow that can restore the surrounding environment by using double-frame or multi-frame point cloud data to obtain the motion information (including two-dimensional / three-dimensional velocity) of each position point in the environment. The scene flow is essentially a three-dimensional form of optical flow, and the calculation of the scene flow can be converted to the calculation of the optical flow after converting the three-dimensional point cloud structure into a two-dimensional feature in the BEV (Bird's Eye View, bird's eye view). That is, the two-dimensional scene flow in this step can also be referred to as optical flow.

[0049] The motion head estimation network can be a neural network, such as Motion Head. After obtaining the target feature map, the target feature map can be input into the motion head estimation network for motion estimation processing to process the target feature map into a two-dimensional scene flow.

[0050] The two-dimensional scene flow includes the motion information of each point in the above environment. Each point herein represents each position in the environment, so the motion information of each point herein can also be the motion information of each position in the environment. The motion information herein can be velocity information, such as two-dimensional motion information, including velocity in the X direction and velocity in the Y direction.

[0051] After obtaining the motion information of each position in the environment, the environment herein includes the vehicle and the objects around the vehicle, i.e., the motion information of each position on the vehicle and the motion information of each position around the vehicle can also be obtained. In this way, the motion information of each position on the vehicle and the motion information of each position around the vehicle can be used to provide a more accurate and comprehensive motion information for the autonomous vehicle to assist the vehicle to better and more accurately perform autonomous driving.

[0052] In the motion information estimation method, a pseudo image feature map corresponding to the original point cloud data is obtained, the pseudo image feature map is input into a preset motion estimation network for motion estimation processing, a target feature map corresponding to the pseudo image feature map is determined, and then a scene flow corresponding to the original point cloud data is determined according to the target feature map; wherein the scene flow includes motion information of each point in the environment corresponding to the original point cloud data, and the motion estimation network includes multiple motion estimation layers, and the estimation result of any motion estimation layer in the multiple motion estimation layers is used as the input of the adjacent next motion estimation layer. In this method, when estimating the motion information in the scene flow through the motion estimation network including the multiple motion estimation layers, the estimation result of any motion estimation layer can be used as the input of the adjacent next motion estimation layer, so that the motion estimation of the next layer can be assisted by the motion estimation result of the previous layer, and the information combined during estimation is more abundant, so that the target feature map estimated is more accurate, and then the motion information estimated through the target feature map is more accurate, thereby ensuring the accuracy of the finally estimated motion information.

[0053] The following embodiments further illustrate the specific architecture of the motion estimation network and the process of using the motion estimation network to specifically estimate motion.

[0054] In another embodiment, another motion information estimation method is provided, each motion estimation layer includes a time convolution module and / or a spatial convolution module, and on the basis of the above embodiment, as shown in Figure 5 The S204 can include the following steps:

[0055] S302, input the pseudo image feature map into the motion estimation network, and use the spatial convolution module in each motion estimation layer to perform spatial information convolution processing on the pseudo image feature map to obtain a first feature map of different scales.

[0056] In this step, the motion estimation network includes multiple time convolution modules and multiple spatial convolution modules, and the multiple time convolution modules and the multiple spatial convolution modules can be used as convolution modules in each motion estimation layer in the motion estimation network. The time convolution module can perform time information convolution and other integration processing on the input data, and the spatial convolution module can perform spatial information convolution and other integration processing on the input data.

[0057] After the pseudo image feature map is obtained, the pseudo image feature map can be input from the bottom of the motion estimation network into the motion estimation network, and each spatial convolution module is used to perform spatial information integration processing of different scales on the pseudo image feature map. Here, the feature map output by the previous spatial convolution module can be used as the input of the next spatial convolution module. Each spatial convolution module can perform two-dimensional convolution (for example, down-sampling processing) on the corresponding input data. After the processing of each spatial convolution module, the scale of the output feature map is reduced to half of the input feature map, the time dimension length of the output is unchanged, and the feature dimension length of the output can be doubled to the length of the input feature dimension. In this way, feature maps of different scales can be obtained, which can be denoted as first scale feature maps.

[0058] For example, the pseudo image feature map can be represented as T*C*W*H, and there are three spatial convolution modules. The input of the first spatial convolution module is the pseudo image feature map, and the output can be T*2C*W / 2*H / 2. The input of the second spatial convolution module is T*2C*W / 2*H / 2, and the output can be T*4C*W / 4*H / 4. The input of the third spatial convolution module is T*4C*W / 4*H / 4, and the output can be T*8C*W / 8*H / 8. T can be the time dimension, C can be the feature dimension, W can be the length of the two-dimensional plane, and H can be the width of the two-dimensional plane.

[0059] In S304, the time convolution module in each motion estimation layer is used to perform time information convolution processing on the first feature map of the corresponding scale to obtain second feature maps of different scales.

[0060] In this step, as mentioned above, the pseudo image feature map can be first input into the spatial convolution module in each motion estimation layer for convolution information integration processing to obtain the first feature map of the corresponding scale. Then, the time convolution module in the corresponding motion estimation layer can be used to perform time information integration processing on the corresponding first feature map, that is, to perform convolution processing on the input data in the time dimension to obtain the second feature map of the corresponding scale.

[0061] The convolution kernel size here can be (T, 1, 1), that is, the time dimension T of the second feature map output by each time convolution module is changed to 1, and the length and width remain unchanged. The length of the output feature dimension is the same as the length of the input feature dimension.

[0062] In addition, as described above, the spatial convolution module and the time convolution module in the embodiment are independent of each other, that is, the number of spatial convolution modules and the number of time convolution modules can not be equal.

[0063] S306, determining a target feature map corresponding to the pseudo image feature map according to the second feature maps of different scales.

[0064] In this step, after obtaining the second feature maps of different scales, the target feature map can be finally obtained by further processing the second feature maps of different scales, such as convolution processing, upsampling processing, etc.

[0065] It should be noted that the length and width of the target feature map are generally the same as the length and width of the input pseudo image feature map, that is, the length and width of the input pseudo image feature map are not changed.

[0066] In this embodiment, the pseudo image feature map is input into the spatial convolution module in each layer of the motion estimation layer for spatial information convolution processing, and the time convolution module is used to perform time information convolution processing on the feature map output by the corresponding spatial convolution module to obtain the target feature map. In this way, the spatial dimension and time dimension of the pseudo image feature map are processed by the mutually independent spatial convolution module and time convolution module, respectively, which can reduce the calculation amount of the motion estimation network and improve the efficiency of motion information estimation.

[0067] The following embodiment describes the process of determining the target feature map from the second feature maps of different scales output by the time convolution module.

[0068] In another embodiment, another motion information estimation method is provided, which is based on the above-mentioned embodiment, as shown in Figure 6 S306 can include the following steps:

[0069] S402, upsampling processing is performed on the second feature map located at the top layer of the motion estimation layer in each second feature map to obtain an upsampled feature map.

[0070] In this step, referring to the architecture block diagram of the motion estimation network shown in Figure 7 The motion estimation network shown in the figure is an inverted pyramid structure, the bottom layer or bottom of the motion estimation network is the top layer or top of the pyramid, and the top layer or top of the motion estimation network is the bottom layer or bottom of the pyramid. It should be noted that Figure 7 In this embodiment, the motion estimation network can be a network including any number of layers, such as a network including two layers, a network including three layers, a network including five layers, etc.

[0071] In Figure 7In the above, S-block represents a spatial convolution module, T-block represents a temporal convolution module, Upsample represents an up-sampling module, and Concat represents concatenation. As can be seen, the pseudo image feature map can be input from the bottom of the motion estimation network to the motion estimation network, and the spatial information of each spatial convolution module S-block in the motion estimation network is convoluted to obtain a first feature map of different scales. Then, the scale map output by each spatial convolution module is convoluted by a corresponding temporal convolution module to obtain a second feature map of a corresponding scale. It should be noted that when the pseudo image feature map is input from the bottom of the motion estimation network to the motion estimation network, the pseudo image feature map is also convoluted by a temporal convolution module to obtain a second feature map of a corresponding scale.

[0072] Further, for the second feature map obtained by each temporal convolution module, as an optional embodiment, the scale of the second feature map located at the top motion estimation layer in each second feature map is smaller than the scale of the second feature map located at the bottom motion estimation layer. Similarly, for the first feature map obtained by each spatial convolution module, as an optional embodiment, the scale of the first feature map located at the top motion estimation layer in each first feature map is smaller than the scale of the first feature map located at the bottom motion estimation layer.

[0073] That is, each spatial convolution module can be located in the down-sampling stage of the motion estimation network, and each temporal convolution module can be located in the up-sampling stage of the motion estimation network, so that the convolution processing in the time dimension can be performed from the top layer of the motion estimation network, so that the data in the adjacent time range and spatial range can be referred to in each motion estimation layer, and a more accurate motion estimation result can be obtained.

[0074] From the top layer of the motion estimation network, after the corresponding second feature map is obtained by the temporal convolution module of the top layer, the second feature map obtained by the temporal convolution module of the top layer can be up-sampled to obtain an up-sampled feature map. The up-sampling processing is generally performed in a manner of 2 raised to the power, that is, each up-sampling processing is to double the size of the input data. The up-sampling processing is mainly to facilitate the subsequent feature connection and convolution processing.

[0075] S404, a convolution operation is performed, which includes: concatenating and convoluting the up-sampled feature map and the second feature map in the adjacent next motion estimation layer to obtain an intermediate feature map.

[0076] In this step, as mentioned above, the second feature map obtained by the top layer temporal convolution module can be up-sampled to obtain an up-sampled feature map, and the up-sampled feature map can be concatenated and convoluted with the second feature map in the adjacent next motion estimation layer to obtain an intermediate feature map. Figure 1Generally, the scale of the up-sampling feature map is the same as the scale of the second feature map output by the adjacent next layer of the time convolution module, so here the up-sampling feature map and the second feature map output by the adjacent next layer of the time convolution module can be connected or spliced (for example Figure 7 Concat in the above formula), to obtain a connected or spliced feature map. Then the spliced feature map can be subjected to convolution processing, and the convolution processing here can also be performed by the spatial convolution module S-block to obtain an intermediate feature map.

[0077] S406, whether the iteration stopping condition is met is detected, if not, the intermediate feature map is subjected to up-sampling processing to obtain a new up-sampling feature map, and the above convolution operation is iteratively performed until the iteration stopping condition is met, to obtain a target feature map.

[0078] In this step, taking the second layer as an example, after obtaining the intermediate feature map of the second layer, it can be detected whether the iteration stopping condition is met, if so, the iteration process of the motion estimation network is ended, and the intermediate feature map of the second layer obtained at present is taken as the target feature map. If not, the intermediate feature map of the second layer is subjected to up-sampling processing to obtain an up-sampling feature map corresponding to the intermediate feature map of the second layer, and the up-sampling feature map is taken as a new up-sampling feature map, and the above step of connecting and convoluting the up-sampling feature map and the second feature map in the adjacent next layer of the motion estimation layer is returned to execute, to obtain the intermediate feature map of the third layer.

[0079] Then it can be continuously detected whether the iteration stopping condition is met, if so, the iteration process of the motion estimation network is ended, and the intermediate feature map of the third layer obtained at present is taken as the target feature map. If not, the intermediate feature map of the third layer is subjected to up-sampling processing in the above manner to obtain a new up-sampling feature map, and then the above step of connecting and convoluting the up-sampling feature map and the second feature map in the adjacent next layer of the motion estimation layer is iteratively executed to obtain the intermediate feature map, until the iteration stopping condition is met at present. Here, when the iteration condition is met at present, the intermediate feature map obtained at present can be taken as the final target feature map.

[0080] The iteration stopping condition can include at least one of the following: the scale of the intermediate feature map is the same as the scale of the pseudo image feature map; the iteration number reaches an iteration number threshold.

[0081] Based on this, the above detection of whether the iteration stop condition is met at present can be detection of whether the scale of the current obtained intermediate feature map is the same as the scale of the original pseudo image feature map. If they are the same, it means that the iteration stop condition is met. If they are not the same, it means that the iteration stop condition is not met, and iteration processing needs to be continued. Here, the scale being the same mainly refers to the length and width of the two-dimensional plane being the same.

[0082] Or it can also be detected whether the current iteration number is greater than the iteration number threshold. If the current iteration number is less than or equal to the iteration number threshold, it means that the iteration stop condition is not met, and iteration processing needs to be continued. If the current iteration number is greater than the iteration number threshold, it means that the iteration stop condition is met.

[0083] In this embodiment, the convolution operation is iteratively performed after the upsampling processing of the top layer second feature map. The specific iterative convolution operation includes connecting and convolving the upsampling feature map and the second feature map of the adjacent next layer to obtain an intermediate feature map, and obtaining a target feature map when the iteration stop condition is met. In this way, through continuous iteration operation, the time and space information of the adjacent layer can be combined when each layer performs motion estimation, the combined information is more and richer, and no deconvolution and temporal pooling processing is needed. This can reduce the operation amount of the network for motion estimation, and also can ensure the accuracy of the estimation result obtained by the final motion estimation. At the same time, by setting multiple different iteration stop conditions, the entire iteration process can be more perfect, and a dead loop can be avoided, so that an accurate motion estimation result can be obtained smoothly.

[0084] The following embodiments describe a possible implementation of obtaining a corresponding pseudo image feature map from the original point cloud data.

[0085] In another embodiment, another motion information estimation method is provided, which is based on the above embodiment, as shown in Figure 8 The S202 can include the following steps:

[0086] S502, obtaining original point cloud data.

[0087] In this step, the acquisition method of the original point cloud data can refer to the related acquisition method in S202 described above, which will not be repeated here.

[0088] S504, inputting the original point cloud data into a preset column feature network for column feature conversion processing to determine a pseudo image feature map.

[0089] In this step, the column feature network can be a neural network, for example, it can be Pillar Feature Net, which mainly converts the original point cloud data into a pseudo image.

[0090] In the specific conversion, first, the point cloud is divided into regions, which is similar to voxel, and the biggest difference between the constructed pillar and voxel is that the pillar has no height limit in the z direction. The points in the pillar also need to be augmented from the original 4-dimensional information to 9-dimensional (the additional 5 dimensions are the three coordinates of the center point of the pillar and the distance from each point x, y to the center). Due to the uneven distribution of the point cloud, a sampling is performed on the non-empty pillar to limit the number of points, and a tensor (tensor) with a size of (D, P, N) is obtained: D is the dimension of the feature, which is 9 here; P is the number of non-empty pillars in each sampling; N is the number of points in each pillar. Next, a feature extraction is performed on the tensor with a size of (D, P, N) to generate a new tensor with a size of (C, P, N), and then a max operation is performed on the channel to reduce the dimension to a tensor with a size of (C, P). Finally, scatter back to the original pillar location to construct a pseudo picture with a size of (C, H, W), where H is the height, W is the width, and C is the number of feature dimensions.

[0091] In addition, the above is the conversion of the point cloud data at one time point to the pseudo picture, and the time information also needs to be considered in the embodiment. Therefore, after the point cloud data at each time point is converted into a pseudo picture, they are arranged in time sequence, and a pseudo picture including time information is obtained. Here, each pseudo picture can be a two-dimensional picture.

[0092] As mentioned above, the pseudo picture includes multiple types of features, which is essentially a feature map, so the pseudo picture including time information obtained above is the pseudo image feature map mentioned above.

[0093] Here, by converting the original point cloud data into a two-dimensional pseudo image feature map including time information, the subsequent 2D convolution can be performed to realize motion estimation, which can improve the efficiency of subsequent motion estimation.

[0094] In the embodiment, the obtained point cloud data is input into the pillar feature network to perform pillar feature conversion processing, and the pseudo image feature map is determined, so that the subsequent 2D convolution can be performed to realize motion estimation, which improves the efficiency of subsequent motion estimation.

[0095] Next, an embodiment of the present disclosure will be described in combination with a specific autonomous driving scene, which can be described in combination with the flow structure block diagram shown in Figure 9 On the basis of the above embodiment, the method comprises the following steps:

[0096] S1, obtaining original point cloud data (i.e., Point cloud in the figure);

[0097] S2, inputting the original point cloud data into a preset pillar feature network for pillar feature conversion processing to determine a pseudo image feature map;

[0098] Specifically, the pillar feature network (i.e., Pillar Feature Net in the figure) can first convert the original point cloud data (i.e., Point cloud in the figure) into stacked pillars (i.e., stacked pillars in the figure), then perform feature learning (i.e., Learned Features in the figure), and finally perform pseudo image conversion processing (i.e., Pseudo image in the figure) to obtain the final converted pseudo image feature map.

[0099] S3, inputting the pseudo image feature map into a motion estimation network (i.e., Neck in the figure) for motion estimation processing to determine a target feature map corresponding to the pseudo image feature map;

[0100] S4, inputting the target feature map into a motion head estimation network (i.e., Motion Head in the figure) for processing to obtain a prediction result (i.e., Predictions in the figure); the prediction result is a scene flow corresponding to the original point cloud data, and the scene flow includes motion information of each point in the environment corresponding to the original point cloud data.

[0101] Further, the three networks of the pillar feature network, the motion estimation network, and the motion head estimation network can be jointly trained using training samples, which include a plurality of training point cloud data and real motion information corresponding to each training point cloud data. Specifically, in the training process, the three networks can be jointly trained by the loss between the predicted motion information of each training point cloud data and the corresponding real motion information.

[0102] It should be understood that although each step in the above flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, at least part of the steps in the above figure can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0103] In one embodiment, as shown in Figure 10 A motion information estimation apparatus is provided, comprising: an acquisition module, a motion estimation module and a determination module, wherein:

[0104] The acquisition module is configured to acquire a pseudo-image feature map corresponding to the original point cloud data.

[0105] The motion estimation module is configured to input the pseudo-image feature map into a preset motion estimation network for motion estimation processing to determine a target feature map corresponding to the pseudo-image feature map; wherein the motion estimation network comprises a plurality of motion estimation layers, and the estimation result of any one of the plurality of motion estimation layers is taken as the input of the adjacent next motion estimation layer.

[0106] The determination module is configured to determine a scene flow corresponding to the original point cloud data according to the target feature map; the scene flow comprises motion information of each point in the environment corresponding to the original point cloud data.

[0107] Optionally, the pseudo-image feature map comprises four-dimensional features, and the four dimensions are respectively a time dimension, a feature dimension, a length of a two-dimensional plane and a width of the two-dimensional plane.

[0108] In another embodiment, another motion information estimation apparatus is provided, and on the basis of the above embodiment, each motion estimation layer comprises a time convolution module and / or a spatial convolution module, and the motion estimation module can comprise:

[0109] The spatial estimation unit is configured to input the pseudo-image feature map into the motion estimation network, and perform spatial information convolution processing on the pseudo-image feature map by using the spatial convolution module in each motion estimation layer to obtain first feature maps of different scales.

[0110] The time estimation unit is configured to perform time information convolution processing on the first feature maps of the corresponding scales by using the time convolution module in each motion estimation layer to obtain second feature maps of different scales.

[0111] The target feature map determination unit is configured to determine the target feature map corresponding to the pseudo-image feature map according to the second feature maps of different scales.

[0112] In another embodiment, another motion information estimation apparatus is provided, and on the basis of the above embodiment, the target feature map determination unit can comprise:

[0113] The up-sampling sub-unit is configured to perform up-sampling processing on the second feature map located at the top motion estimation layer in each second feature map to obtain an up-sampled feature map.

[0114] The execution subunit is configured to perform a convolution operation, which comprises connecting and performing convolution processing on the up-sampling feature map and a second feature map in a next motion estimation layer adjacent to the up-sampling feature map, to obtain an intermediate feature map.

[0115] The detection subunit is configured to detect whether an iteration stop condition is met, and if not, perform up-sampling processing on the intermediate feature map to obtain a new up-sampling feature map, and iteratively perform the convolution operation until the iteration stop condition is met, to obtain a target feature map.

[0116] Optionally, the iteration stop condition comprises at least one of the following: the scale of the intermediate feature map is the same as the scale of the pseudo-image feature map; and the number of iterations reaches an iteration threshold.

[0117] Optionally, the scale of the second feature map located at the top motion estimation layer is smaller than the scale of the second feature map located at the bottom motion estimation layer.

[0118] In another embodiment, another motion information estimation device is provided, and on the basis of the above-mentioned embodiment, the determination module can comprise:

[0119] The determination unit is configured to input the target feature map into a preset motion head estimation network for processing, to obtain a scene flow corresponding to the original point cloud data; the scene flow is a two-dimensional scene flow.

[0120] In another embodiment, another motion information estimation device is provided, and on the basis of the above-mentioned embodiment, the acquisition module can comprise:

[0121] The acquisition unit is configured to acquire the original point cloud data.

[0122] The conversion unit is configured to input the original point cloud data into a preset column feature network for column feature conversion processing, to determine the pseudo-image feature map.

[0123] For specific limitations of the motion information estimation device, refer to the limitations of the motion information estimation method in the foregoing, which will not be repeated here. Each module in the motion information estimation device can be realized by software, hardware, and a combination thereof, in whole or in part. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so as to be called and executed by a processor to perform the operations corresponding to each module.

[0124] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as a memory including instructions, is also provided. The instructions can be executed by a processor of a computer device to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0125] In an example embodiment, a computer program product is also provided. The computer program is executed by a processor to implement the above method. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, some or all of the above method can be implemented according to the processes or functions described in the embodiments of the present disclosure.

[0126] A person of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Any reference to a memory, storage, database, or other medium used in the embodiments of the present disclosure can include at least one of a non-volatile and volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disc, a flash memory, or an optical memory. The volatile memory can include a random access memory (RAM) or an external cache memory. As an illustration but not limitation, the RAM can be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM).

[0127] Any combination of the technical features of the above embodiments can be made. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present disclosure.

[0128] The above embodiments only express several implementation manners of the present disclosure, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present disclosure, some modifications and improvements can be made, which are within the scope of the present disclosure. Therefore, the scope of protection of the patent of the present disclosure should be subject to the appended claims.

Claims

1. A motion information estimation method characterized by comprising: The method comprises: obtaining a pseudo-image feature map corresponding to the original point cloud data; inputting the pseudo-image feature map into a preset motion estimation network for motion estimation processing to determine a target feature map corresponding to the pseudo-image feature map; wherein the motion estimation network comprises multiple motion estimation layers, and the estimation result of any one of the multiple motion estimation layers is taken as the input of the next motion estimation layer; determining a scene flow corresponding to the original point cloud data according to the target feature map; the scene flow comprises motion information of each point in the environment corresponding to the original point cloud data. The method comprises:

2. The method of claim 1, wherein, inputting the pseudo-image feature map into the motion estimation network, performing spatial information convolution processing on the pseudo-image feature map by using a spatial convolution module in the motion estimation layer to obtain first feature maps of different scales; performing time information convolution processing on the first feature maps of the corresponding scales by using a time convolution module in the motion estimation layer to obtain second feature maps of different scales; and determining the target feature map corresponding to the pseudo-image feature map according to the second feature maps of different scales. The method comprises: performing up-sampling processing on the second feature map located at the top motion estimation layer in each of the second feature maps to obtain an up-sampled feature map; performing convolution operation, which comprises connecting and performing convolution processing on the up-sampled feature map and the second feature map in the next motion estimation layer adjacent to the up-sampled feature map to obtain an intermediate feature map; 3. The method of claim 2, wherein, detecting whether an iteration stop condition is met, and if not, performing up-sampling processing on the intermediate feature map to obtain a new up-sampled feature map, and iteratively performing the convolution operation until the iteration stop condition is met to obtain the target feature map. The iteration stop condition comprises at least one of the following: the scale of the intermediate feature map is the same as the scale of the pseudo-image feature map; 4. The method according to any one of claims 1 to 3, characterized in that, the number of iterations reaches an iteration threshold.

5. The method according to any one of claims 1 to 3, characterized in that, The scale of the second feature map located at the top motion estimation layer in each of the second feature maps is smaller than the scale of the second feature map located at the bottom motion estimation layer. The method comprises:

6. The method according to any one of claims 1 to 3, characterized in that, inputting the target feature map into a preset motion head estimation network for processing to obtain a scene flow corresponding to the original point cloud data; and the scene flow is a two-dimensional scene flow. The method comprises: obtaining original point cloud data; 7. The method according to any one of claims 1 to 3, characterized in that, inputting the original point cloud data into a preset column feature network for column feature conversion processing to determine the pseudo-image feature map.

8. A motion information estimation apparatus characterized by comprising: The pseudo-image feature map comprises four-dimensional features, and the four dimensions are time dimension, feature dimension, length of a two-dimensional plane and width of a two-dimensional plane. The device comprises: an obtaining module configured to obtain a pseudo-image feature map corresponding to original point cloud data; a motion estimation module, configured to input the pseudo-image feature map into a preset motion estimation network for motion estimation processing to determine a target feature map corresponding to the pseudo-image feature map; wherein the motion estimation network comprises a plurality of motion estimation layers, and an estimation result of any one of the plurality of motion estimation layers is taken as an input of a next motion estimation layer adjacent to the any one of the plurality of motion estimation layers; a determination module, configured to determine a scene flow corresponding to the original point cloud data according to the target feature map; the scene flow comprising motion information of each point in an environment corresponding to the original point cloud data; wherein the inputting of the pseudo-image feature map into the preset motion estimation network for motion estimation processing to determine the target feature map corresponding to the pseudo-image feature map comprises: inputting the pseudo-image feature map into the motion estimation network, performing spatial information convolution processing on the pseudo-image feature map by using a spatial convolution module in the motion estimation layer to obtain a first feature map of different scales; performing time information convolution processing on the first feature map of a corresponding scale by using a time convolution module in the motion estimation layer to obtain a second feature map of different scales; and determining the target feature map corresponding to the pseudo-image feature map according to the second feature map of different scales.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1-7.

10. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method in any one of claims 1-7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1-7. The computer program is executed by the processor to implement the steps of the method in any one of claims 1-7.

Images