Motion information determination method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to a motion information determination method and device, equipment, a storage medium and a program product. The method comprises the following steps: controlling a laser radar to perform target detection to obtain at least two frames of point cloud data; determining first motion information of the laser radar in a reference coordinate system and second motion information of a to-be-detected target object relative to the laser radar according to the two frames of point cloud data; and determining target motion information of the target object in the reference coordinate system according to the first motion information and the second motion information. According to the method, the laser radar can be used for measuring the motion information of the target object, so that the application scene of the laser radar is expanded.

Description

[0001] This application is a divisional application of a Chinese patent application with an application date of May 23, 2022, an application number of 202210563625.3, and an invention name of “Method, device, equipment, and storage medium for determining motion information”. Technical Field

[0002] The embodiments of the present disclosure relate to the technical field of target detection, and in particular, to a method, apparatus, device, storage medium, and program product for determining motion information. Background Art

[0003] With the development of automobile technology, autonomous driving technology has emerged. LiDAR is a commonly used sensor in autonomous driving technology. It mainly uses the principle of time of flight, that is, by measuring the time difference between the emission and return of the laser beam, the distance from the laser radar to the surface of the object is calculated.

[0004] Based on the above principles, most lidars can only measure the geometric information and surface reflectivity information of an object, but it is difficult to measure the motion information of the object. Summary of the invention

[0005] The embodiments of the present disclosure provide a method, apparatus, device, storage medium and program product for determining motion information, which can utilize a laser radar to measure the motion information of a target object, thereby expanding the applicable scenarios of the laser radar.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for determining motion information, the method comprising:

[0007] Control the laser radar to detect targets and obtain at least two frames of point cloud data;

[0008] Determine, according to the two frames of point cloud data, first motion information of the laser radar in a reference coordinate system and second motion information of a target object to be detected relative to the laser radar;

[0009] Determining target motion information of the target object in the reference coordinate system according to the first motion information and the second motion information;

[0010] Wherein, determining the first motion information of the laser radar in the reference coordinate system and the second motion information of the target object to be detected relative to the laser radar according to the two frames of point cloud data includes:

[0011] For the first point cloud data and the second point cloud data, coordinate conversion processing is performed respectively to obtain a first depth map corresponding to the first point cloud data and a second depth map corresponding to the second point cloud data; wherein the coordinate conversion processing is a processing of converting three-dimensional coordinates into two-dimensional coordinates;

[0012] Determine first motion information according to the marker points corresponding to the first depth map and the second depth map;

[0013] The second motion information is determined by performing optical flow calculation according to the first depth map and the second depth map.

[0014] In a second aspect, an embodiment of the present disclosure provides a device for determining motion information, the device comprising:

[0015] A data acquisition module is used to control the laser radar to detect targets and obtain at least two frames of point cloud data;

[0016] A first information determination module is used to determine first motion information of the laser radar in the reference coordinate system and second motion information of the target object to be detected relative to the laser radar based on two frames of point cloud data;

[0017] A second information determination module, used to determine target motion information of the target object in the reference coordinate system according to the first motion information and the second motion information;

[0018] Among them, the first information determination module is also used for: performing coordinate conversion processing on the first point cloud data and the second point cloud data, respectively, to obtain a first depth map corresponding to the first point cloud data and a second depth map corresponding to the second point cloud data; wherein the coordinate conversion processing is the processing of converting three-dimensional coordinates into two-dimensional coordinates; determining the first motion information according to the marker points corresponding to the first depth map and the second depth map; and determining the second motion information by performing optical flow calculation based on the first depth map and the second depth map.

[0019] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method of the first aspect is implemented when the processor executes the computer program.

[0020] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method of the first aspect described above is implemented.

[0021] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which implements the method of the first aspect when executed by a processor.

[0022] The method, device, equipment, storage medium and program product for determining motion information provided by the embodiments of the present disclosure control the laser radar to detect a target and obtain at least two frames of point cloud data; determine the first motion information of the laser radar in the reference coordinate system and the second motion information of the target object to be detected relative to the laser radar based on the two frames of point cloud data; determine the target motion information of the target object in the reference coordinate system based on the first motion information and the second motion information. The embodiments of the present disclosure can determine the motion information of the target object through the motion information of the laser radar itself and the motion information of the target object relative to the laser radar, thereby realizing the functional expansion of the laser radar and making the laser radar applicable to more scenarios.

[0023] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0025] Figure 1 A diagram of an application environment of a method for determining motion information in an embodiment;

[0026] Figure 2 is a flowchart of a method for determining motion information in one embodiment;

[0027] Figure 3 FIG2 is a second flow chart of a method for determining motion information in an embodiment;

[0028] Figure 4 A schematic diagram of a flow chart of a step of determining first motion information and second motion information in an embodiment;

[0029] Figure 5 is a schematic diagram of a process of converting a depth map in one embodiment;

[0030] Figure 6 is a schematic diagram of a flow chart of a step of determining first motion information in an embodiment;

[0031] Figure 7 is a schematic diagram of a flow chart of a step of determining second motion information in an embodiment;

[0032] Figure 8is a flow chart of a method for determining motion information in another embodiment;

[0033] Fig. 9 is one of the structural block diagrams of a device for determining motion information in an embodiment;

[0034] Fig.10 This is a second structural block diagram of a device for determining motion information in an embodiment;

[0035] Fig.11 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0036] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure more clear, the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present disclosure and are not used to limit the embodiments of the present disclosure.

[0038] First of all, before specifically introducing the technical solution of the embodiment of the present disclosure, the technical background or technical evolution context on which the embodiment of the present disclosure is based is introduced. LiDAR is a commonly used sensor in autonomous driving technology. It mainly adopts the principle of time of flight, that is, by measuring the time difference between the emission and return of the laser beam, the distance from the laser radar to the surface of the object is calculated. Based on the above principle, most laser radars can only measure the geometric information and surface reflectivity information of the object, but it is difficult to measure the motion information of the object.

[0039] In the scheme for determining motion information provided by the embodiments of the present disclosure, the laser radar is controlled to perform target detection to obtain at least two frames of point cloud data; based on the two frames of point cloud data, the first motion information of the laser radar in the reference coordinate system and the second motion information of the target object to be detected relative to the laser radar are determined; based on the first motion information and the second motion information, the target motion information of the target object in the reference coordinate system is determined. The embodiments of the present disclosure can determine the motion information of the target object through the motion information of the laser radar itself and the motion information of the target object relative to the laser radar, thereby realizing the functional expansion of the laser radar and making the laser radar suitable for more scenarios. It should be noted that the applicant has made a lot of creative efforts to find that it is difficult for the laser radar to measure the motion information of the object and the technical solutions described in the following embodiments.

[0040] The technical solutions involved in the embodiments of the present disclosure are introduced below in combination with the scenarios to which the embodiments of the present disclosure are applied.

[0041] The method for determining motion information provided by the embodiment of the present disclosure can be applied to Figure 1 In the application environment shown.

[0042] The application environment includes a vehicle 102, in which an electronic device and a laser radar are arranged. The laser radar can sense the surrounding environment of the vehicle, and the electronic device performs target detection based on the point cloud data collected by the laser radar, and controls the vehicle according to the detection result to realize automatic driving of the vehicle.

[0043] In one embodiment, Figure 2 As shown, a method for determining motion information is provided, and the method is applied to Figure 1 The electronic device in the example is used to illustrate, and the following steps are included:

[0044] Step 201, control the laser radar to perform target detection and obtain at least two frames of point cloud data.

[0045] The electronic device communicates with the laser radar, controls the laser radar to emit laser pulses to the surrounding environment and detects the returned laser pulses to obtain point cloud data. When the laser radar detects continuously, it can obtain at least two frames of point cloud data.

[0046] Step 202, based on the two frames of point cloud data, determine the first motion information of the laser radar in the reference coordinate system and the second motion information of the target object to be detected relative to the laser radar.

[0047] The motion information includes at least one of speed information, acceleration information and direction information.

[0048] After acquiring the point cloud data, the first motion information of the laser radar in the reference coordinate system and the second motion information of the target object to be detected relative to the laser radar can be determined according to the difference between the two frames of point cloud data.

[0049] For example, the reference coordinate system is the world coordinate system, and the target object to be detected is a pedestrian. Based on the difference between the two frames of point cloud data, the position change of the laser radar in the world coordinate system can be determined, thereby determining the speed and direction of the laser radar, and obtaining the first motion information of the laser radar. At the same time, based on the two frames of point cloud data, the position change of the pedestrian relative to the laser radar can be determined, thereby determining the speed and direction of the pedestrian, and obtaining the second motion information of the pedestrian.

[0050] The embodiment of the present disclosure does not limit the method for determining the first motion information and the second motion information, and they can be selected according to actual conditions.

[0051] Optionally, the two frames of point cloud data are two adjacent frames of point cloud data.

[0052] When determining the first motion information and the second motion information, point cloud data of two adjacent frames may be used. For example, point cloud data at time T and time T-1 may be used. It is understandable that since the time interval between the two frames of point cloud data is short, the first motion information and the second motion information may be determined more accurately.

[0053] Optionally, the two frames of point cloud data are non-adjacent frames of point cloud data. In this case, the speed information can be determined based on the position change and the time interval between the two frames of point cloud data. The disclosed embodiment does not limit the time interval between the two frames of point cloud data.

[0054] Step 203: Determine target motion information of the target object in the reference coordinate system according to the first motion information and the second motion information.

[0055] After determining the first motion information and the second motion information, the motion of the target object in the reference coordinate system can be determined based on the motion of the lidar in the reference coordinate system indicated by the first motion information and the motion of the target object relative to the lidar, thereby determining the target motion information of the target object.

[0056] For example, determining the motion of the lidar in the world coordinate system and the motion of the pedestrian relative to the lidar can determine the motion of the pedestrian in the world coordinate system.

[0057] In the above-mentioned method for determining motion information, the laser radar is controlled to perform target detection to obtain at least two frames of point cloud data; based on the two frames of point cloud data, the first motion information of the laser radar in the reference coordinate system and the second motion information of the target object to be detected relative to the laser radar are determined; based on the first motion information and the second motion information, the target motion information of the target object in the reference coordinate system is determined. The disclosed embodiment can determine the motion information of the target object through the motion information of the laser radar itself and the motion information of the target object relative to the laser radar, thereby realizing the functional expansion of the laser radar and making the laser radar applicable to more scenarios.

[0058] In one embodiment, the process of determining the target motion information of the target object in the reference coordinate system based on the first motion information and the second motion information may include: determining the target motion information of each point cloud point on the target object in the reference coordinate system based on the first motion information of the laser radar in the reference coordinate system and the second motion information of each point cloud point on the target object relative to the laser radar.

[0059] In practical applications, the point cloud data obtained by the laser radar when detecting the target object is obtained by irradiating the laser pulse to the target object and being reflected by the target object. Each point cloud point in the point cloud data corresponds to the irradiation point on the target object. Therefore, the motion information of the target object relative to the laser radar includes the second motion information of each point cloud point on the target object relative to the laser radar.

[0060] When determining the target motion information of the target object, the motion of each point cloud point on the target object in the reference coordinate system can be determined based on the motion of the laser radar in the reference coordinate system and the motion of each point cloud point on the target object relative to the laser radar, thereby obtaining the target motion information of each point cloud point on the target object in the reference coordinate system.

[0061] In the above embodiment, the target motion information of each point cloud point on the target object in the reference coordinate system is determined based on the first motion information of the laser radar in the reference coordinate system and the second motion information of each point cloud point on the target object relative to the laser radar. The target motion information of each point cloud point on the target object is determined in the embodiment of the present disclosure, which can more accurately reflect the motion of the target object.

[0062] In one embodiment, Figure 3 As shown, before determining the target motion information of each point cloud point on the target object in the reference coordinate system, the embodiment of the present disclosure may further include:

[0063] Step 204: interpolate the first motion information to obtain a plurality of interpolated third motion information.

[0064] The third motion information corresponds one-to-one to the point cloud points on the target object.

[0065] When a mechanical laser radar emits laser pulses, it rotates mechanically, so the collection time of each point cloud point is not the same. In this case, the first motion information can be interpolated to obtain multiple third motion information, so that each point cloud point on the target object has a corresponding third motion information.

[0066] The interpolation process may include: calculating an average value of adjacent data and using the average value as an interpolation value. The interpolation process may be performed in other ways, which are not limited in the embodiments of the present disclosure.

[0067] Correspondingly, step 203 includes: determining the target motion information of each point cloud point on the target object in the reference coordinate system according to the second motion information of each point cloud point on the target object and the third motion information corresponding to each point cloud point.

[0068] After the interpolation process, each point cloud point on the target object has a corresponding third motion information, so that for the target object, the target motion information can be obtained point by point according to the third motion information and the second motion information.

[0069] It is understandable that when determining the target motion information point by point, preferably each point cloud point has a corresponding third motion information after the interpolation process, and in order to reduce the amount of calculation of the interpolation process, multiple point cloud points can also correspond to the same third motion information after the interpolation process. The embodiments of the present disclosure do not limit this and can be set according to actual conditions.

[0070] Since the target motion information of the target object includes the target motion information of multiple point cloud points, the target motion information may also be referred to as scene flow.

[0071] In the above embodiment, the first motion information is interpolated to obtain a plurality of third motion information after interpolation; the target motion information of each point cloud point on the target object in the reference coordinate system is determined according to the second motion information of each point cloud point on the target object and the third motion information corresponding to each point cloud point. The disclosed embodiment first interpolates the first motion information, and then determines the target motion information of the target object point by point, so that the target motion information can be more accurate.

[0072] In one embodiment, Figure 4 As shown, the process of determining the first motion information of the laser radar in the reference coordinate system and the second motion information of the target object to be detected relative to the laser radar based on the two frames of point cloud data may include:

[0073] Step 301 : performing coordinate transformation processing on first point cloud data and second point cloud data respectively to obtain a first depth map corresponding to the first point cloud data and a second depth map corresponding to the second point cloud data.

[0074] The coordinate conversion process is a process of converting three-dimensional coordinates into two-dimensional coordinates.

[0075] For two frames of point cloud data, each point cloud point in the first point cloud data is converted from a three-dimensional coordinate system to a two-dimensional coordinate system to obtain a first depth map; each point cloud point in the second point cloud data is converted from a three-dimensional coordinate system to a two-dimensional coordinate system to obtain a second depth map.

[0076] Understandably, the depth map can not only retain more three-dimensional spatial information, but also is more suitable for learning by the neural network model, so it can obtain a larger receptive field and more global information.

[0077] Step 302: Determine first motion information according to the marker points corresponding to the first depth map and the second depth map.

[0078] The position of the same marker point in the first depth map and the second depth map is determined. According to the position change of the marker point, the movement of the laser radar can be determined to obtain the first movement information of the laser radar.

[0079] For example, if the landmark point is the tip of a tree, the position change of the lidar can be inferred based on the position change of the tree tip in the first depth map and the second depth map; then, based on the position change of the lidar and the time interval between the first point cloud data and the second point cloud data, the speed information, direction information, etc. of the lidar can be determined to obtain the first motion information of the lidar.

[0080] It can be understood that by determining the first motion information based on the marker points corresponding to the first depth map and the second depth map, the effect of a laser odometer can be achieved.

[0081] Step 303: Perform optical flow calculation according to the first depth map and the second depth map to determine second motion information.

[0082] The position of the target object in the first depth map and the second depth map is determined, and optical flow calculation is performed according to the position of the target object in the first depth map and the position of the target object in the second depth map, so as to determine the second motion information of the target object.

[0083] In the above embodiment, coordinate transformation processing is performed on the first point cloud data and the second point cloud data to obtain a first depth map corresponding to the first point cloud data and a second depth map corresponding to the second point cloud data; the first motion information is determined based on the landmarks corresponding to the first depth map and the second depth map; and the second motion information is determined by performing optical flow calculation based on the first depth map and the second depth map. In the process of determining the first motion information and the second motion information, the disclosed embodiment first converts the point cloud data into a depth map. Since the depth map can not only retain more three-dimensional spatial information, but is also more suitable for learning by a neural network model, a larger receptive field and more global information can be obtained, and the accuracy of the motion information can be further improved.

[0084] In one embodiment, Figure 5 As shown, the process of performing coordinate transformation processing on the first point cloud data and the second point cloud data to obtain a first depth map corresponding to the first point cloud data and a second depth map corresponding to the second point cloud data may include the following steps:

[0085] Step 3011, for the first point cloud data and the second point cloud data, convert the three-dimensional coordinates of each point cloud point in the reference coordinate system into the spherical coordinate system to obtain the first angle and the second angle of each point cloud point in the spherical coordinate system.

[0086] The first angle is the angle between the point cloud point and the x-axis of the spherical coordinate system; the second angle is the angle between the point cloud point and the z-axis of the spherical coordinate system.

[0087] The above-mentioned reference coordinate system may be a world coordinate system. For point cloud point 1 in the first point cloud data, its position in the world coordinate system is (x1, y1, z1). Point cloud point 1 can be converted to the spherical coordinate system according to the coordinate conversion relationship between the world coordinate system and the spherical coordinate system, and the position of point cloud point 1 in the spherical coordinate system is obtained as where r1 is the distance between the position of the point cloud point in the spherical coordinate system and the origin of the spherical coordinate system, θ1 is the first angle, and is the second angle. Similarly, coordinate conversion is performed on other point cloud points in the first point cloud data and each point cloud point in the second point cloud data to obtain the first angle and the second angle of each point cloud point in the spherical coordinate system.

[0088] Step 3012: Use the first angle as the horizontal coordinate, the second angle as the vertical coordinate, and the surface reflectivity information and / or depth information of the point cloud point as the pixel value to obtain a first depth map and a second depth map.

[0089] For each point cloud point in the first point cloud data, the first angle θ1 is used as the horizontal coordinate, the second angle is used as the vertical coordinate, and then the surface reflectivity information of the point cloud point is used as the pixel value, then the first depth map corresponding to the first point cloud data can be obtained. Alternatively, for each point cloud point in the first point cloud data, the first angle θ1 is used as the horizontal coordinate, the second angle is used as the vertical coordinate, and then the surface reflectivity information of the point cloud point is used as the pixel value of one channel, and the depth information of the point cloud point is used as the pixel value of another channel, then the first depth map corresponding to the first point cloud data can be obtained. The second depth map corresponding to the second point cloud data can be obtained in the same manner as above.

[0090] The above pixel values ​​may also adopt prior information such as projection error and classification result. The embodiment of the present disclosure does not limit the pixel values ​​and may be set according to actual conditions.

[0091] It can be understood that, in the first depth map and the second depth map, each row of pixels corresponds to a laser scanning line (beam); each pixel in each row corresponds to a scanning point of the laser radar.

[0092] In the above embodiment, for the first point cloud data and the second point cloud data, the three-dimensional coordinates of each point cloud point in the reference coordinate system are converted to the spherical coordinate system to obtain the first angle and the second angle of each point cloud point in the spherical coordinate system; the first angle is used as the horizontal coordinate, the second angle is used as the vertical coordinate, and the surface reflectivity information and / or depth information of the point cloud point is used as the pixel value to obtain the first depth map and the second depth map. The disclosed embodiment converts the point cloud data into a depth map, which can not only realize the conversion of coordinates, but also the pixel value of the depth map can also include surface reflectivity information, depth information, etc. Therefore, the depth map can not only retain more three-dimensional spatial information, but also obtain a larger receptive field and more global information.

[0093] In one embodiment, Figure 6 As shown, the process of determining the first motion information according to the marker points corresponding to the first depth map and the second depth map may include the following steps:

[0094] Step 3021: Determine the landmark points corresponding to the first depth map and the second depth map.

[0095] The first depth map and the second depth map may be subjected to feature extraction to obtain a first feature map and a second feature map. Afterwards, the first feature map and the second feature map are classified and identified to obtain the category of each target object in the first feature map and the category of each target object in the second feature map. Next, according to the category of the target object, the same target object is selected, and a certain point on the target object is used as a landmark point.

[0096] For example, after extracting features and classifying the first depth map and the second depth map, it is determined that the target objects include vehicles, pedestrians, and trees. Since vehicles and pedestrians are in motion, trees that are stationary are selected and the treetops are determined as landmarks.

[0097] In practical applications, other methods may be used to determine corresponding landmark points in the first depth map and the second depth map, which is not limited in the embodiments of the present disclosure.

[0098] Step 3022: Determine a first position of the marker point in the first depth map and a second position of the marker point in the second depth map.

[0099] After the corresponding marker points in the first depth map and the second depth map are determined, a first position of the marker point in the first depth map and a second position of the marker point in the second depth map may be determined.

[0100] For example, a first position of the tree tip in the first depth map and a second position of the marker in the second depth map are determined.

[0101] Step 3023: determine first motion information according to the first position and the second position.

[0102] According to the internal and external parameters of the laser radar and the first position of the marker point in the first depth map, the position of the laser radar at the previous moment can be determined; according to the internal parameters of the laser radar and the first position of the marker point in the second depth map, the position of the laser radar at the next moment can be determined; according to the position change of the laser radar at the previous moment and the next moment, the first motion information such as the speed information and direction information of the laser radar can be determined according to the time interval between the first point cloud data and the second point cloud data.

[0103] In the above embodiment, the marker points corresponding to the first depth map and the second depth map are determined; the first position of the marker point in the first depth map and the second position of the marker point in the second depth map are determined; and the first motion information is determined based on the first position and the second position. The embodiment of the present disclosure determines the first motion information based on the corresponding marker points, thereby achieving the effect of a laser odometer.

[0104] In an embodiment, the process of determining the second motion information by performing optical flow calculation based on the first depth map and the second depth map may include: performing optical flow calculation on the first depth map and the second depth map using a block matching algorithm to obtain the second motion information.

[0105] The block matching algorithm is an algorithm in motion estimation, which can quickly and accurately calculate the optical flow, thereby obtaining the second motion information of the target object.

[0106] like Figure 7 As shown, the process of calculating the optical flow using the block matching algorithm may include the following steps:

[0107] Step 3031: randomly initialize each first pixel in the first depth map to obtain initial optical flow information of each first pixel.

[0108] The initial optical flow information includes initial speed information and direction information.

[0109] For each first pixel in the first depth map, random initialization is first performed so that each first pixel has corresponding initial optical flow information.

[0110] Step 3032: Determine, according to the initial optical flow information of each first pixel, the second pixel corresponding to each first pixel in the second depth map.

[0111] According to the initial optical flow information, the position of each first pixel after being changed according to the initial optical flow information can be determined, and the position is matched with the second depth map to determine the second pixel corresponding to the first pixel in the second depth map.

[0112] For example, the first pixel (1, 1) in the first depth map is changed to (1, 2) according to the initial optical flow information, and then the second pixel at position (1, 2) in the second depth map is determined to correspond to the first pixel (1, 1).

[0113] Step 3033: If the first pixel matches the second pixel corresponding to the first pixel, it is determined that the initial optical flow information of the first pixel is a feasible solution for the first pixel.

[0114] The pixel value similarity between the first pixel and the second pixel can be calculated. If the pixel value similarity is greater than a preset threshold, it is determined that the first pixel matches the second pixel, and then the initial optical flow information of the first pixel is determined as a feasible solution for the first pixel. If the first pixel does not match the second pixel, the first pixel can obtain a feasible solution propagated by adjacent pixels, re-determine the corresponding second pixel, and determine again whether it matches.

[0115] In practical applications, other methods may be used to determine whether the first pixel matches the second pixel, which is not limited in the embodiments of the present disclosure.

[0116] Step 3034: perform spatial propagation on the feasible solutions of each first pixel to obtain second motion information.

[0117] After the first pixel determines the feasible solution, the feasible solutions of each first pixel are spatially propagated. For example, the first pixel (1, 1) propagates the feasible solution to the first pixel (1, 2) and the first pixel (2, 1), etc. The first pixel (1, 2) can also propagate the feasible solution to each first pixel adjacent to itself. During the process of spatial propagation of the feasible solution, if the convergence condition is met, the feasible solutions under the convergence condition are summarized to obtain the second motion information.

[0118] The above-mentioned convergence condition may include that the number of propagation times reaches a preset number, etc., which is not limited in the embodiments of the present disclosure.

[0119] Optionally, the feasible solution can be propagated across the boundary. For example, the first pixel (1, 1) is the left boundary of the first row of pixels in the first depth map, and (1, n) is the left boundary of the first row of pixels in the first depth map. The first pixel (1, 1) can propagate the feasible solution to (1, n) across the boundary, and the first pixel (1, n) can also propagate the feasible solution to (1, 1) across the boundary.

[0120] In the above embodiment, each first pixel in the first depth map is randomly initialized to obtain the initial optical flow information of each first pixel; based on the initial optical flow information of each first pixel, the second pixel corresponding to each first pixel in the second depth map is determined; if the first pixel matches the second pixel corresponding to the first pixel, the initial optical flow information of the first pixel is determined to be a feasible solution for the first pixel; the feasible solution of each first pixel is spatially propagated to obtain the second motion information. The disclosed embodiment uses a block matching algorithm to quickly and accurately determine the second motion information, thereby improving the detection speed and detection efficiency of target detection.

[0121] In one embodiment, Figure 8 As shown, a method for determining motion information is provided, and the method is applied to Figure 1 The electronic device in the example is used to illustrate, and the following steps are included:

[0122] Step 401, control the laser radar to perform target detection and obtain at least two frames of point cloud data.

[0123] Step 402 : performing coordinate transformation processing on the first point cloud data and the second point cloud data, respectively, to obtain a first depth map corresponding to the first point cloud data and a second depth map corresponding to the second point cloud data.

[0124] In one embodiment, for the first point cloud data and the second point cloud data, the three-dimensional coordinates of each point cloud point in the reference coordinate system are converted into the spherical coordinate system to obtain the first angle and the second angle of each point cloud point in the spherical coordinate system. The first angle is the angle between the point cloud point and the x-axis of the spherical coordinate system; the second angle is the angle between the point cloud point and the z-axis of the spherical coordinate system. The first angle is used as the horizontal coordinate, the second angle is used as the vertical coordinate, and the surface reflectivity information and / or depth information of the point cloud point is used as the pixel value to obtain the first depth map and the second depth map.

[0125] Step 403: determine first motion information according to the marker points corresponding to the first depth map and the second depth map.

[0126] In one of the embodiments, a marker point corresponding to the first depth map and the second depth map is determined; a first position of the marker point in the first depth map and a second position of the marker point in the second depth map are determined; and first motion information is determined based on the first position and the second position.

[0127] Step 404: perform optical flow calculation on the first depth map and the second depth map using a block matching algorithm to obtain second motion information.

[0128] In one of the embodiments, each first pixel in the first depth map is randomly initialized to obtain initial optical flow information of each first pixel; based on the initial optical flow information of each first pixel, the second pixel corresponding to each first pixel in the second depth map is determined; if the first pixel matches the second pixel corresponding to the first pixel, the initial optical flow information of the first pixel is determined to be a feasible solution for the first pixel; and the feasible solutions of each first pixel are spatially propagated to obtain second motion information.

[0129] Step 405: interpolate the first motion information to obtain a plurality of interpolated third motion information.

[0130] The third motion information corresponds one-to-one to the point cloud points on the target object.

[0131] Step 406 , determining the target motion information of each point cloud point on the target object in the reference coordinate system according to the second motion information of each point cloud point on the target object and the third motion information corresponding to each point cloud point.

[0132] In the above embodiment, the laser radar is controlled to detect the target and obtain at least two frames of point cloud data; the first point cloud data and the second point cloud data are respectively converted into coordinates to obtain the first depth map and the second depth map; the first motion information of the laser radar and the second motion information of the target object are determined according to the first depth map and the second depth map; the first motion information is interpolated, and then the target motion information of the target object is determined point by point. The disclosed embodiment can detect the motion information of the target object by using the laser radar, which expands the function of the laser radar and makes the laser radar applicable to more scenarios.

[0133] It should be understood that although Figures 2 to 8 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figures 2 to 8 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0134] In one embodiment, Fig. 9 As shown, a device for determining motion information is provided, comprising:

[0135] The data acquisition module 501 is used to control the laser radar to detect the target and obtain at least two frames of point cloud data;

[0136] A first information determination module 502 is used to determine first motion information of the laser radar in the reference coordinate system and second motion information of the target object to be detected relative to the laser radar according to the two frames of point cloud data;

[0137] The second information determination module 503 is used to determine the target motion information of the target object in the reference coordinate system according to the first motion information and the second motion information.

[0138] In one embodiment, the second information determination module 503 is specifically used to determine the target motion information of each point cloud point on the target object in the reference coordinate system based on the first motion information of the laser radar in the reference coordinate system and the second motion information of each point cloud point on the target object relative to the laser radar.

[0139] In one embodiment, if Fig.10 As shown, it also includes:

[0140] An interpolation module 504 is used to perform interpolation processing on the first motion information to obtain a plurality of third motion information after interpolation; wherein the third motion information corresponds to the point cloud points on the target object one by one;

[0141] Correspondingly, the second information determination module 503 is specifically used to determine the target motion information of each point cloud point on the target object in the reference coordinate system according to the second motion information of each point cloud point on the target object and the third motion information corresponding to each point cloud point.

[0142] In one of the embodiments, the first information determination module 502 is specifically used to perform coordinate conversion processing on the first point cloud data and the second point cloud data, respectively, to obtain a first depth map corresponding to the first point cloud data and a second depth map corresponding to the second point cloud data; wherein the coordinate conversion processing is the processing of converting three-dimensional coordinates into two-dimensional coordinates; determining the first motion information according to the landmark points corresponding to the first depth map and the second depth map; and determining the second motion information by performing optical flow calculation based on the first depth map and the second depth map.

[0143] In one embodiment, the first information determination module 502 is specifically configured to perform optical flow calculation on the first depth map and the second depth map by using a block matching algorithm to obtain the second motion information.

[0144] In one of the embodiments, the first information determination module 502 is specifically used to randomly initialize each first pixel in the first depth map to obtain initial optical flow information of each first pixel; determine the second pixel corresponding to each first pixel in the second depth map based on the initial optical flow information of each first pixel; if the first pixel matches the second pixel corresponding to the first pixel, determine that the initial optical flow information of the first pixel is a feasible solution for the first pixel; and perform spatial propagation on the feasible solutions of each first pixel to obtain second motion information.

[0145] In one embodiment, the first information determination module 502 is specifically used to determine the marker points corresponding to the first depth map and the second depth map; determine the first position of the marker point in the first depth map and the second position of the marker point in the second depth map; and determine the first motion information based on the first position and the second position.

[0146] In one embodiment, the first information determination module 502 is specifically used to convert the three-dimensional coordinates of each point cloud point in the reference coordinate system into the spherical coordinate system for the first point cloud data and the second point cloud data, and obtain the first angle and the second angle of each point cloud point in the spherical coordinate system; the first angle is the angle between the point cloud point and the x-axis of the spherical coordinate system; the second angle is the angle between the point cloud point and the z-axis of the spherical coordinate system; the first angle is used as the horizontal coordinate, the second angle is used as the vertical coordinate, and the surface reflectivity information and / or depth information of the point cloud point is used as the pixel value to obtain the first depth map and the second depth map.

[0147] In one embodiment, the two frames of point cloud data are two adjacent frames of point cloud data.

[0148] For the specific definition of the device for determining motion information, please refer to the definition of the method for determining motion information above, which will not be repeated here. Each module in the above-mentioned device for determining motion information can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0149] Fig.11 1 is a block diagram of an electronic device 1300 according to an exemplary embodiment. For example, the electronic device 1300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0150] Reference Fig.11, the electronic device 1300 may include one or more of the following components: a processing component 1302, a memory 1304, a power component 1306, a multimedia component 1308, an audio component 1310, an input / output (I / O) interface 1312, a sensor component 1314, and a communication component 1316. The memory stores computer programs or instructions running on the processor.

[0151] The processing component 1302 generally controls the overall operation of the electronic device 1300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1302 may include one or more processors 1320 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 1302 may include one or more modules to facilitate the interaction between the processing component 1302 and other components. For example, the processing component 1302 may include a multimedia module to facilitate the interaction between the multimedia component 1308 and the processing component 1302.

[0152] The memory 1304 is configured to store various types of data to support operations on the electronic device 1300. Examples of such data include instructions for any application or method operating on the electronic device 1300, contact data, phone book data, messages, pictures, videos, etc. The memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0153] The power supply component 1306 provides power to the various components of the electronic device 1300. The power supply component 1306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1300.

[0154] The multimedia component 1308 includes a touch display screen that provides an output interface between the electronic device 1300 and the user. In some embodiments, the touch display screen may include a liquid crystal display (LCD) and a touch panel (TP). The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1308 includes a front camera and / or a rear camera. When the electronic device 1300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0155] The audio component 1310 is configured to output and / or input audio signals. For example, the audio component 1310 includes a microphone (MIC), and when the electronic device 1300 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1304 or sent via the communication component 1316. In some embodiments, the audio component 1310 also includes a speaker for outputting audio signals.

[0156] I / O interface 1312 provides an interface between processing component 1302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0157] The sensor assembly 1314 includes one or more sensors for providing various aspects of status assessment for the electronic device 1300. For example, the sensor assembly 1314 can detect the open / closed state of the electronic device 1300, the relative positioning of components, such as the display and keypad of the electronic device 1300, and the sensor assembly 1314 can also detect the position change of the electronic device 1300 or a component of the electronic device 1300, the presence or absence of user contact with the electronic device 1300, the orientation or acceleration / deceleration of the electronic device 1300, and the temperature change of the electronic device 1300. The sensor assembly 1314 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1314 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0158] The communication component 1316 is configured to facilitate wired or wireless communication between the electronic device 1300 and other devices. The electronic device 1300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0159] In an exemplary embodiment, the electronic device 1300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned method for determining motion information.

[0160] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1304 including instructions, and the instructions can be executed by a processor 1320 of the electronic device 1300 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0161] In an exemplary embodiment, a computer program product is also provided, and when the computer program is executed by a processor, the above method can be implemented. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, part or all of the above method can be implemented in whole or in part according to the process or function of the embodiment of the present disclosure.

[0162] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the embodiments of the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0163] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The above embodiments only express several implementation methods of the embodiments of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the embodiments of the present disclosure, and these all belong to the protection scope of the embodiments of the present disclosure. Therefore, the protection scope of the patent of the embodiments of the present disclosure shall be based on the attached claims.

Claims

1. A method for determining motion information, It is characterized in that include: Control the laser radar to detect targets and obtain at least two frames of point cloud data; Determine, according to the two frames of point cloud data, first motion information of the laser radar in a reference coordinate system and second motion information of a target object to be detected relative to the laser radar; Determining target motion information of the target object in the reference coordinate system according to the first motion information and the second motion information; Wherein, determining the first motion information of the laser radar in the reference coordinate system and the second motion information of the target object to be detected relative to the laser radar according to the two frames of point cloud data includes: For the first point cloud data and the second point cloud data, coordinate conversion processing is performed respectively to obtain a first depth map corresponding to the first point cloud data and a second depth map corresponding to the second point cloud data; wherein the coordinate conversion processing is a process of converting three-dimensional coordinates into two-dimensional coordinates; Determine the first motion information according to the marker points corresponding to the first depth map and the second depth map; The second motion information is determined by performing optical flow calculation according to the first depth map and the second depth map.

2. The method according to claim 1, It is characterized in that The determining the second motion information by performing optical flow calculation according to the first depth map and the second depth map includes: Optical flow calculation is performed on the first depth map and the second depth map using a block matching algorithm to obtain the second motion information.

3. The method according to claim 2, It is characterized in that The using a block matching algorithm to perform optical flow calculation on the first depth map and the second depth map to obtain the second motion information includes: Randomly initializing each first pixel in the first depth map to obtain initial optical flow information of each first pixel; Determine, according to the initial optical flow information of each of the first pixels, a second pixel corresponding to each of the first pixels in the second depth map; If the first pixel matches the second pixel corresponding to the first pixel, determining that the initial optical flow information of the first pixel is a feasible solution for the first pixel; The feasible solutions of each of the first pixels are spatially propagated to obtain the second motion information.

4. The method according to claim 1, It is characterized in that The determining the first motion information according to the marker points corresponding to the first depth map and the second depth map includes: Determining landmark points corresponding to the first depth map and the second depth map; Determining a first position of the marker point in the first depth map and a second position of the marker point in the second depth map; The first motion information is determined according to the first position and the second position.

5. The method according to claim 1, It is characterized in that The step of performing coordinate transformation processing on the first point cloud data and the second point cloud data to obtain a first depth map corresponding to the first point cloud data and a second depth map corresponding to the second point cloud data includes: For the first point cloud data and the second point cloud data, the three-dimensional coordinates of each point cloud point in the reference coordinate system are converted into a spherical coordinate system to obtain a first angle and a second angle of each point cloud point in the spherical coordinate system; the first angle is the angle between the point cloud point and the x-axis of the spherical coordinate system; the second angle is the angle between the point cloud point and the z-axis of the spherical coordinate system; The first angle is used as the abscissa, the second angle is used as the ordinate, and the surface reflectivity information and / or depth information of the point cloud points are used as pixel values ​​to obtain the first depth map and the second depth map.

6. The method according to any one of claims 1 to 5, It is characterized in that The two frames of point cloud data are two adjacent frames of point cloud data.

7. A device for determining motion information, It is characterized in that The device includes: A data acquisition module is used to control the laser radar to detect targets and obtain at least two frames of point cloud data; A first information determination module is used to determine first motion information of the laser radar in a reference coordinate system and second motion information of a target object to be detected relative to the laser radar based on two frames of point cloud data; A second information determination module, configured to determine target motion information of the target object in the reference coordinate system according to the first motion information and the second motion information; Among them, the first information determination module is also used for: performing coordinate conversion processing on the first point cloud data and the second point cloud data, respectively, to obtain a first depth map corresponding to the first point cloud data and a second depth map corresponding to the second point cloud data; wherein the coordinate conversion processing is a process of converting three-dimensional coordinates into two-dimensional coordinates; determining the first motion information according to the landmark points corresponding to the first depth map and the second depth map; and determining the second motion information by performing optical flow calculation based on the first depth map and the second depth map.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When a processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A storage medium having a computer program stored thereon, It is characterized in that The computer program implements the steps of the method of any one of claims 1 to 6 when executed by a processor.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images