Target motion estimation method, device and computer storage medium

Abstract

The present application discloses a method, device and computer storage medium for estimating target motion, which includes: obtaining fusion data of two consecutive frames of multiple sensors; processing the fusion data using deep learning technology to obtain depth images of two consecutive frames of binocular images, initial segmentation labels of two consecutive frames of binocular images and optical flow images of two consecutive frames of binocular images respectively; constructing an objective function, using three-dimensional point cloud data, depth images of two consecutive frames of binocular images and optical flow images of two consecutive frames of binocular images to calculate the initial target motion in the objective function; constructing an energy function, using the initial segmentation label and the initial target motion to initialize the target segmentation label and target motion in the energy function, so that the energy function iteratively optimizes the target segmentation label and target motion, and obtains the predicted target motion trajectory. The above scheme improves the accuracy of target motion estimation.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a method and device for estimating target motion and a computer storage medium. Background Art

[0002] With the development of robot technology, more and more robot products are entering people's lives. The perception technology of dynamic targets is an important guarantee for supporting robots to perform tasks in dynamic environments. In the prior art, robots use background detection and recognition technology and optical flow technology to realize target motion detection and target motion estimation of fixed cameras under static perspectives and ideal scenes, but the above method is limited to target motion detection and target motion estimation of fixed cameras under static perspectives and ideal scenes. If it is applied to mobile platforms, such as target motion estimation in robot operation scenarios, it will lead to inaccurate target motion estimation in robot operation scenarios. Summary of the invention

[0003] The present application at least provides a method, device and computer storage medium for estimating target motion, which can improve the accuracy of target motion estimation.

[0004] A first aspect of the present application provides a method for estimating target motion, the method comprising:

[0005] Acquire two consecutive frames of multi-sensor fusion data, wherein the fusion data includes three-dimensional point cloud data of the laser radar and a binocular image of the binocular camera;

[0006] The fused data is processed by using deep learning technology to obtain depth images of the two consecutive binocular images, initial segmentation labels of the two consecutive binocular images, and optical flow images of the two consecutive binocular images;

[0007] Constructing an objective function, and calculating an initial target motion in the objective function using the three-dimensional point cloud data, the depth image of the two consecutive binocular images, and the optical flow image of the two consecutive binocular images;

[0008] An energy function is constructed, and the target segmentation label and the target motion in the energy function are initialized using the initial segmentation label of the two consecutive binocular images and the initial target motion in the target function, so that the energy function iteratively optimizes the target segmentation label and the target motion to obtain a predicted target motion trajectory.

[0009] A second aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the target motion estimation method in the first aspect.

[0010] A third aspect of the present application provides a computer storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the method for estimating target motion in the first aspect described above is implemented.

[0011] The above scheme obtains two consecutive frames of multi-sensor fusion data, wherein the fusion data includes three-dimensional point cloud data of the lidar sensor and the binocular image of the binocular camera; uses deep learning technology to process the fusion data, and obtains depth images of two consecutive frames of binocular images, initial segmentation labels of targets of two consecutive frames of binocular images, and optical flow images of two consecutive frames of binocular images respectively; constructs an objective function, and calculates the initial target motion in the objective function using the three-dimensional point cloud data, the depth images of two consecutive frames of binocular images, and the optical flow images of two consecutive frames of binocular images; constructs an energy function, and initializes the target segmentation label and target motion in the energy function using the initial segmentation label of the target in two consecutive frames of binocular images and the initial target motion in the objective function, so that the energy function iteratively optimizes the target segmentation label and target motion to obtain the predicted target motion trajectory. The initial segmentation label and initial target motion of two consecutive frames of binocular images can be calculated using the acquired three-dimensional point cloud data of the lidar and the binocular image of the binocular camera, and an energy function can be constructed. The initial segmentation label and the initial target motion are used to initialize the target segmentation label and target motion in the energy function, so that the target motion and the target segmentation label in the energy function can be optimized at the same time, and a high-precision target motion trajectory is obtained, thereby improving the estimation accuracy of the target motion trajectory.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0014] Figure 1 is a flowchart of a first embodiment of a method for estimating target motion provided by the present application;

[0015] Figure 2 is a flow chart of a second embodiment of a method for estimating target motion provided by the present application;

[0016] Figure 3 It is a schematic diagram of the process of obtaining the initial target motion in the target motion estimation method provided by the present application;

[0017] Figure 4 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0018] Figure 5is a schematic diagram of a framework of another embodiment of an electronic device provided by the present application;

[0019] Figure 6 It is a schematic diagram of the framework of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0020] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0021] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0022] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0023] This application proposes a target motion estimation method, which can be applied to estimate the target motion in a dynamic environment when a robot performs a task. The target motion estimation method of this application can improve the accuracy of target motion estimation. For details, please refer to Figure 1 , Figure 1 1 is a flow chart of the first embodiment of the target motion estimation method provided by the present application. The target motion estimation method of the present embodiment can be applied to a target motion estimation device or a server with data processing capability. The present application takes the target motion estimation device as an example for explanation.

[0024] Specifically, the target motion estimation method of this embodiment includes the following steps:

[0025] S101: Acquire two consecutive frames of multi-sensor fusion data.

[0026] In this embodiment, the target motion estimation device, such as an autonomous navigation robot, needs to accurately locate and predict the behavior of other traffic participants in its own operating scene when performing tasks, so as to avoid problems such as collisions between the autonomous navigation robot and other traffic participants when performing tasks.

[0027] Specifically, the multi-sensor includes a laser radar and a binocular camera. The laser radar obtains three-dimensional point cloud data in the current motion environment. The binocular camera obtains binocular images in the current running scene. The binocular images include left images and right images. The fused data includes the three-dimensional point cloud data of the laser radar and the binocular images of the binocular camera. The target motion estimation device of this embodiment can obtain two consecutive frames of three-dimensional point cloud data and binocular image data in the current running scene through the laser radar and binocular camera installed in the target motion estimation device. The fused data is obtained by setting the laser radar and binocular camera in the target motion estimation device, and can be obtained when the target motion estimation device starts to perform a task.

[0028] Among them, the laser radar and the binocular camera can be installed on one side of the running direction of the target motion estimation device. In this embodiment, the laser radar and the binocular camera are installed in the upper and middle part of one side of the running direction of the target motion estimation device, so as to obtain the three-dimensional point cloud data and binocular image in front of the running direction of the target motion estimation device.

[0029] S102: Processing the fused data using deep learning technology to obtain depth images of two consecutive binocular image frames, initial segmentation labels of two consecutive binocular image frames, and optical flow images of two consecutive binocular image frames.

[0030] In order to accurately estimate the target motion trajectory in the fused data of two consecutive frames, the target motion estimation device of this embodiment uses deep learning technology to calculate the initial value of the fused data, and the initial value includes the depth image of two consecutive frames of binocular images, the initial segmentation label of two consecutive frames of binocular images, and the optical flow image of two consecutive frames of binocular images.

[0031] Specifically, since the fused data of two consecutive frames have a dense and stable parallax effect in the deep neural network, the target motion estimation device can use the parallax effect of the two consecutive frames of fused data to obtain a depth image of the two consecutive frames of fused data in the binocular camera coordinate system.

[0032] Since the segmentation result guarantees the accuracy of target motion estimation, in order to accurately estimate the trajectory of target motion, the target motion estimation device of this embodiment can use the segmentation network to perform initial segmentation on the target motion in two consecutive frames of binocular images to obtain an initial segmentation label with low segmentation accuracy.

[0033] In order to improve the efficiency of target motion estimation, the target motion estimation device of this embodiment uses a deep network to calculate the optical flow images of two consecutive frames of binocular images.

[0034] S103: constructing an objective function, and calculating an initial target motion in the objective function using the three-dimensional point cloud data, the depth images of two consecutive binocular image frames, and the optical flow images of two consecutive binocular image frames.

[0035] Based on the depth images of the two consecutive frames of binocular images obtained in S102 , the initial segmentation labels of the targets of the two consecutive frames of binocular images, and the optical flow images of the two consecutive frames of binocular images, the initial target motion in the two consecutive frames of binocular images is roughly estimated.

[0036] Specifically, the target motion estimation device of this embodiment constructs the target function and estimates the initial target motion in the target function using the depth image of two consecutive binocular image frames, the initial segmentation labels of two consecutive binocular image frames, and the optical flow images of two consecutive binocular image frames.

[0037] S104: constructing an energy function, using the initial segmentation labels of two consecutive binocular images and the initial target motion in the target function to initialize the target segmentation label and the target motion in the energy function, so that the energy function iteratively optimizes the target segmentation label and the target motion to obtain a predicted target motion trajectory.

[0038] Since the segmentation label is the guarantee of the target motion estimation accuracy, in order to improve the estimation accuracy of the target motion, the target motion estimation device of this embodiment uses the initial segmentation label and the initial target motion to initialize the target segmentation label and target motion in the energy function, so that the energy function simultaneously iteratively optimizes the target motion and the target segmentation label in the energy function, and obtains a high-precision predicted target motion trajectory, thereby improving the estimation accuracy of the target motion trajectory.

[0039] In the above scheme, the target motion estimation device obtains two consecutive frames of multi-sensor fusion data, wherein the fusion data includes three-dimensional point cloud data of the lidar sensor and the binocular image of the binocular camera; uses deep learning technology to process the fusion data, and obtains depth images of two consecutive frames of binocular images, initial segmentation labels of two consecutive frames of binocular images, and optical flow images of two consecutive frames of binocular images respectively; constructs an objective function, and calculates the initial target motion in the objective function using the three-dimensional point cloud data, the depth images of two consecutive frames of binocular images, and the optical flow images of two consecutive frames of binocular images; constructs an energy function, and initializes the target segmentation label and target motion in the energy function using the initial segmentation label of two consecutive frames of binocular images and the initial target motion in the objective function, so that the energy function iteratively optimizes the target segmentation label and target motion to obtain the predicted target motion trajectory. The initial segmentation label and initial target motion of two consecutive frames of binocular images can be calculated using the acquired three-dimensional point cloud data of the lidar and the binocular image of the binocular camera, and an energy function can be constructed. The initial segmentation label and the initial target motion are used to initialize the target segmentation label and target motion in the energy function, so that the target motion and the target segmentation label in the energy function can be optimized at the same time, and a high-precision target motion trajectory is obtained, thereby improving the estimation accuracy of the target motion trajectory.

[0040] Please continue reading Figure 2 , Figure 2: is a flow chart of the second embodiment of the method for estimating target motion provided by the present application. Specifically, the method of the embodiment of the present disclosure may include the following steps:

[0041] S201: Acquire two consecutive frames of multi-sensor fusion data.

[0042] The target motion estimation device of this embodiment calibrates the external parameter T between the binocular camera and the laser radar. CamLidar And the intrinsic parameter K of the binocular camera.

[0043] S202: Input the fused data into a disparity network to obtain depth images of two consecutive binocular image frames.

[0044] Since the fused data of two consecutive frames has a dense and stable parallax effect in the deep neural network, the target motion estimation device of this embodiment can use the parallax network to process the fused data to obtain the depth image of two consecutive binocular images.

[0045] Specifically, the disparity network may be a hierarchical pyramid cubic matching network PSM-Net (Pyramid StereoMatching Network). For example, the target motion estimation device uses the hierarchical pyramid cubic matching network to obtain the left image I and the right image I of the binocular image at time t0. 0 With the right image R 0 The corresponding depth image D 0 , the left image I in the binocular image at time t1 1 With the right image R 1 Depth image D 1 .

[0046] S203: Input the fused data into a semantic segmentation network to obtain initial segmentation labels for two consecutive frames of binocular images.

[0047] The target motion estimation device of this embodiment uses a semantic segmentation network to perform semantic segmentation on the target in two consecutive frames of binocular images to obtain an initial segmentation label.

[0048] Specifically, the semantic segmentation network model may be Mask R-CNN. For example, the target motion estimation device uses Mask R-CNN to obtain the left image I in the binocular image at time t0. 0 The initial segmentation label is denoted as

[0049] It should be noted that The pixels marked as 0 in the image are defaulted to the background area, and the remaining foreground objects are marked with labels greater than or equal to 1. Among them, the foreground objects include objects with motion attributes or potential motion attributes.

[0050] S204: Input the fused data into an optical flow prediction network to obtain optical flow images of two consecutive frames of binocular images.

[0051] Since the existing optical flow image calculation method is time-consuming, in order to improve the efficiency of target motion estimation, the target motion estimation device of this embodiment uses an optical flow prediction network to obtain optical flow images of two consecutive frames of binocular images.

[0052] Specifically, the optical flow prediction network may be a PWC-Net. For example, the target motion estimation device uses the PWC-Net to obtain the left image I in the binocular image at time t0. 0 At time t1, the left image I in the binocular image 1 The optical flow image of in, Includes the 2D translation vector (Δu, Δv) of each pixel in the image plane.

[0053] S205: constructing an objective function, and calculating an initial target motion in the objective function using the three-dimensional point cloud data, the depth images of two consecutive binocular image frames, and the optical flow images of two consecutive binocular image frames.

[0054] The target motion estimation device of this embodiment constructs an objective function and calculates the initial target motion in the objective function using the three-dimensional point cloud data obtained in S201 and the depth image of two consecutive binocular images and the optical flow image of two consecutive binocular images obtained in S204.

[0055] Specifically, the initial target motion in the objective function is calculated using the three-dimensional point cloud data, the depth image of two consecutive binocular images, and the optical flow image of two consecutive binocular images. Figure 3 , Figure 3 : is a schematic diagram of the process of obtaining the initial target motion in the target motion estimation method provided by the present application. The process of obtaining the initial target motion in this embodiment may include the following steps:

[0056] S1: Obtain the image pixel coordinates and lidar depth image corresponding to the coordinates of the three-dimensional point cloud data.

[0057] The target motion estimation device of this embodiment converts the coordinates (X x y ) of the three-dimensional point cloud data in the laser radar coordinate system into the coordinates (X x y ) of the three-dimensional point cloud data in the laser radar coordinate system based on the extrinsic parameters between the calibrated binocular camera and the laser radar. l ,Y l ,Z l ) is converted to the coordinates (X c ,Y c ,Z c ) and use the internal parameters of the binocular camera to obtain the image pixel coordinates (u, v) and the laser radar depth image D' corresponding to the 3D point cloud data.

[0058] S2: Calculate the correspondence between the image pixel coordinates of two consecutive frames of three-dimensional point cloud data using the optical flow images of two consecutive frames of binocular images and the image pixel coordinates corresponding to the coordinates of the three-dimensional point cloud data.

[0059] In order to obtain the correspondence between the image pixel coordinates of two consecutive frames of three-dimensional point cloud data so as to calculate the initial target motion according to the correspondence, the target motion estimation device of this embodiment uses the optical flow images of two consecutive frames of binocular images and the image pixel coordinates corresponding to the coordinates of the three-dimensional point cloud data to calculate the correspondence between the image pixel coordinates corresponding to the current frame of three-dimensional point cloud data and the image pixel coordinates corresponding to the next frame of three-dimensional point cloud data in the two frames of binocular images. Among them, the optical flow image includes the translation vector (Δu, Δv) of each pixel point on the binocular image. For example, the image pixel coordinates corresponding to the coordinates of the three-dimensional point cloud data at time t0 are (u0, v0), then the image pixel coordinates corresponding to the coordinates of the three-dimensional point cloud data at time t0 (u0, v0) are projected to the image pixel coordinates corresponding to the coordinates of the three-dimensional point cloud data at time t1 are (u1, v1), (u1, v1) = (u0 + Δu, v0 + Δv).

[0060] S3: Fuse the depth image of the binocular image and the lidar depth image to obtain a depth fusion result.

[0061] In order to improve the accuracy of the depth fusion result, the target motion estimation device of this embodiment performs a parallax check on the depth image of the binocular image and the lidar depth image. If the check passes, the lidar depth image is used as the depth fusion result.

[0062] S4: Construct an objective function and calculate the minimum value of the objective function using the corresponding relationship between the image pixel coordinates of two consecutive frames of three-dimensional point cloud data and the depth fusion result.

[0063] The target motion estimation device of this embodiment constructs an objective function and calculates the minimum value of the objective function using the correspondence between the image pixel coordinates of two consecutive frames of three-dimensional point cloud data obtained in S2 and the depth fusion result obtained in S4.

[0064] Among them, the objective function satisfies the following formula:

[0065]

[0066] Among them, p0 is the image pixel coordinate of the 3D point cloud data at time t0, p1 is the image pixel coordinate of the 3D point cloud data at time t1, The target motion is M obj_i =[R,t] T .

[0067] S5: Calculate the initial target motion using the corresponding value of the minimum value of the objective function.

[0068] In a specific embodiment, the target motion estimation device can solve the centroid positions of p0 and p1 respectively, use the coordinates after removing the centroid as the image pixel coordinates of p0 and p1, and obtain the initial target motion M by SVD decomposition or nonlinear optimization iteration method. obj_i .

[0069] S206: Obtain a binocular camera constraint function, a depth image constraint function, and a segmentation label constraint function, and construct an energy function using the binocular camera constraint function, the depth image constraint function, and the segmentation label constraint function.

[0070] In order to improve the estimation accuracy of target motion, the target motion estimation device of this embodiment uses a binocular camera constraint function, a depth image constraint function and a segmentation label constraint function to construct an energy function.

[0071] Specifically, the energy function satisfies the following formula:

[0072]

[0073] Among them, λ is the weight configuration of the binocular camera constraint function, the depth image constraint function and the segmentation label constraint function, ξ is the predicted target motion trajectory, and S is the target segmentation label.

[0074] The constraint function of the stereo camera satisfies the following formula:

[0075]

[0076] Among them, α p ∈{0,1}, indicating whether the pixel is blocked. The specific occlusion judgment can be obtained based on the change of two consecutive depth images; ρ(x) is the loss function, ρ(x) = (x 2 +ε 2 ) α In a specific embodiment, α = 0.45, ε = 10 -5 ; p and p′ correspond to the image pixel coordinates at time t0 and time t1 respectively. The reverse depth projection of p and p′ under ξ can be expressed as follows:

[0077]

[0078] in, is the back projection function of the binocular camera. Under the pinhole camera model, Satisfy the following formula:

[0079]

[0080] Among them, K contains f x 、f y, cx, and cy are the internal parameters of the camera.

[0081] The depth image constraint function satisfies the following formula:

[0082]

[0083] Among them, the function D(p) is defined as the depth value corresponding to the depth image at point p, D 0 (p) and D 1 (p′) corresponds to the depth image at time t0 and time t1 respectively.

[0084] The segmentation label constraint function satisfies the following formula:

[0085]

[0086] in, The image gradient direction between pq can be determined by bicubic interpolation; N(p) represents the 8-neighborhood of pixel p; γ is the weight coefficient value.

[0087] S207: Initialize the target segmentation label and target motion in the energy function using the initial segmentation label and the initial target motion, fix the initial target motion, iteratively update and optimize the energy function, and obtain the target segmentation label.

[0088] In order to improve the optimization efficiency of the energy function, the target motion estimation device of this embodiment optimizes the energy function in steps. Specifically, the target motion estimation device uses the initial segmentation label and the initial target motion to initialize the target segmentation label and the target motion in the energy function, fixes the initial target motion, uses the initial segmentation label as a candidate value for the energy function segmentation optimization, performs multi-label segmentation, and obtains the target segmentation label.

[0089] In a specific embodiment, QPBO (quadratic pseudo-Boolean optimization) is used to iteratively update the segmentation labels in the energy function.

[0090] S208: Fix the target segmentation label and use the Gauss-Newton method to optimize the energy function to obtain the predicted target motion trajectory.

[0091] Based on the target segmentation label obtained in S207, the target motion estimation device of this embodiment fixes the target segmentation label, uses the initial target motion to initialize the target motion in the energy function, and iteratively optimizes the target motion with the energy function to obtain the predicted target motion trajectory.

[0092] In a specific embodiment, the Gauss-Newton method is used to optimize the first two terms of the energy function, namely, E photo With E depth , accurately obtain the predicted target motion trajectory.

[0093] In order to avoid the influence of the self-motion of the binocular camera on the target motion estimation in a moving scene, the target motion estimation device of this embodiment can remove the segmentation label of the foreground target in the initial segmentation label, and use the remaining segmentation labels of the initial segmentation label to initialize the target segmentation label in the energy function, so that the energy function iteratively updates and optimizes the target segmentation label.

[0094] In the above scheme, the target motion estimation device uses the acquired three-dimensional point cloud data of the lidar and the binocular image of the binocular camera to calculate the initial segmentation label and the initial target motion of two consecutive frames of binocular images, constructs an energy function, and uses the initial segmentation label and the initial target motion to initialize the target segmentation label and the target motion in the energy function, so that the energy iteratively optimizes the target segmentation label and the target motion to obtain the predicted target motion trajectory. The target motion estimation device uses the three-dimensional point cloud data of the laser radar and the binocular image of the binocular camera to estimate the target motion trajectory in a mobile environment, thereby improving the accuracy of the target motion estimation; the target motion estimation device inputs the fusion data into the optical flow prediction network to obtain the optical flow images of two consecutive frames of binocular images respectively, thereby improving the efficiency of the target motion estimation; the target motion estimation device performs parallax verification on the depth image of the binocular image and the laser radar depth image, and if the verification passes, the laser radar depth image is used as the depth fusion result to improve the accuracy of the depth fusion result; the target motion estimation device optimizes the energy function in combination with the semantic attributes of the three-dimensional point cloud data of the laser radar and the binocular image of the binocular camera, which is conducive to optimizing the energy function by specifying the semantic category target; the target motion estimation device optimizes the energy function step by step using the initial segmentation label and the initial target motion, thereby improving the efficiency and accuracy of the energy function optimization; the target motion estimation device removes the segmentation label of the foreground target in the initial segmentation label, and uses the remaining segmentation labels of the initial segmentation label to iteratively initialize the target segmentation label in the energy function, so that the energy function updates and optimizes the target segmentation label, thereby avoiding the influence of the self-motion of the binocular camera on the target motion estimation in the mobile scene.

[0095] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0096] See also Figure 4 , Figure 4 The electronic device includes an acquisition module, a processing module, a first calculation module, and a second calculation module.

[0097] The acquisition module is used to acquire two consecutive frames of multi-sensor fusion data, wherein the fusion data includes three-dimensional point cloud data of the lidar sensor and binocular images of the binocular camera.

[0098] The processing module is used to process the fused data using deep learning technology to obtain depth images of two consecutive frames of binocular images, initial segmentation labels of two consecutive frames of binocular images, and optical flow images of two consecutive frames of binocular images.

[0099] The first calculation module is used to construct the target function, and calculate the initial target motion in the target function using the three-dimensional point cloud data, the depth image of two consecutive frames of binocular images, and the optical flow image of two consecutive frames of binocular images.

[0100] The second calculation module is used to construct an energy function, and optimize the energy function by using the initial segmentation labels of two consecutive binocular images and the initial target motion in the objective function to obtain the predicted target motion trajectory.

[0101] See also Figure 5 , Figure 5 : is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other, and the processor 52 is used to execute program instructions stored in the memory 51 to implement the steps of any of the above-mentioned target motion estimation method embodiments. In a specific implementation scenario, the electronic device 50 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 50 may also include a laptop computer, a tablet computer, a headset, a mobile phone, and other mobile devices, which are not limited here.

[0102] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps of the above-mentioned estimation method embodiment of any target motion. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0103] See also Figure 6 , Figure 6The computer readable storage medium 60 stores program instructions 601 that can be executed by a processor, and the program instructions 601 are used to implement the steps of any of the above-mentioned target motion estimation method embodiments.

[0104] In some embodiments, the functions or modules included in the device provided in this embodiment can be used to execute the method described in the above method embodiment. Its specific implementation can refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0105] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0106] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A method for estimating target motion, characterized in that: The estimation method includes: Acquire two consecutive frames of multi-sensor fusion data, wherein the fusion data includes three-dimensional point cloud data of the laser radar and a binocular image of the binocular camera; The fused data is processed by using deep learning technology to obtain depth images of the two consecutive binocular images, initial segmentation labels of the two consecutive binocular images, and optical flow images of the two consecutive binocular images; The step of constructing an objective function and calculating the initial target motion in the objective function using the three-dimensional point cloud data, the depth image of the two consecutive binocular images, and the optical flow image of the two consecutive binocular images comprises: Obtaining image pixel coordinates and a laser radar depth image corresponding to the coordinates of the three-dimensional point cloud data; Calculating the correspondence between the image pixel coordinates of the two consecutive frames of three-dimensional point cloud data by using the optical flow images of the two consecutive frames of binocular images and the image pixel coordinates corresponding to the coordinates of the three-dimensional point cloud data; Fusing the depth image of the binocular image and the laser radar depth image to obtain a depth fusion result; Constructing the objective function, and calculating the minimum value of the objective function using the corresponding relationship between the image pixel coordinates of the two consecutive frames of three-dimensional point cloud data and the depth fusion result; Calculating the initial target motion using the corresponding value of the minimum value of the objective function; The step of constructing an energy function, initializing the target segmentation label and the target motion in the energy function by using the initial segmentation label of the two consecutive binocular images and the initial target motion in the target function, and making the energy function iteratively optimize the target segmentation label and the target motion to obtain the predicted target motion trajectory includes: Obtaining a binocular camera constraint function, a depth image constraint function, and a segmentation label constraint function, and constructing the energy function using the binocular camera constraint function, the depth image constraint function, and the segmentation label constraint function; Initializing the target segmentation label and the target motion in the energy function by using the initial segmentation label and the initial target motion, fixing the initial target motion, iteratively updating and optimizing the energy function, and obtaining the target segmentation label; The target segmentation label is fixed, and the energy function is optimized using the Gauss-Newton method to obtain the predicted target motion trajectory.

2. The estimation method according to claim 1, characterized in that The step of obtaining the image pixel coordinates and the laser radar depth image corresponding to the coordinates of the three-dimensional point cloud data includes: Based on the external parameters between the binocular camera and the laser radar, the coordinates of the three-dimensional point cloud data in the laser radar coordinate system are converted into coordinates in the binocular camera coordinate system; Based on the internal parameters of the binocular camera, the image pixel coordinates and the laser radar depth image corresponding to the three-dimensional point cloud data are obtained from the coordinates of the three-dimensional point cloud data in the binocular camera coordinate system.

3. The estimation method according to claim 2, characterized in that The optical flow images of two consecutive frames of binocular images include a translation vector of each pixel point on the binocular image; The step of calculating the correspondence between the image pixel coordinates of the two consecutive frames of three-dimensional point cloud data by using the optical flow images of the two consecutive frames of binocular images and the image pixel coordinates corresponding to the coordinates of the three-dimensional point cloud data comprises: Based on the image pixel coordinates and the translation vector of each pixel point on the binocular image, the correspondence between the pixel coordinates corresponding to the three-dimensional point cloud data of the current frame and the pixel coordinates corresponding to the three-dimensional point cloud data of the next frame in two consecutive binocular images is calculated.

4. The estimation method according to claim 1, characterized in that: The step of fusing the depth image of the binocular image and the laser radar depth image to obtain a depth fusion result includes: Performing a parallax check on the depth image of the binocular image and the laser radar depth image to determine whether the check passes; If so, the lidar depth image is used as the depth fusion result.

5. The estimation method according to claim 1, characterized in that: The step of processing the fused data by using deep learning technology to obtain depth images of the two consecutive binocular images, initial segmentation labels of the two consecutive binocular images, and optical flow images of the two consecutive binocular images respectively comprises: Inputting the fused data into a disparity network to obtain depth images of the two consecutive binocular image frames respectively; Inputting the fused data into a semantic segmentation network to obtain initial segmentation labels of the targets of the two consecutive binocular image frames respectively; The fused data is input into an optical flow prediction network to obtain an optical flow image of the two consecutive binocular images.

6. The estimation method according to claim 1, characterized in that: The initial segmentation label includes a segmentation label of a foreground target, and the step of using the initial segmentation label and the initial target motion to initialize the target segmentation label and the target motion in the energy function, fixing the initial target motion, iteratively updating and optimizing the energy function to obtain the target segmentation label includes: The segmentation labels of the foreground targets in the initial segmentation labels are removed, and the target segmentation labels in the energy function are initialized using the remaining segmentation labels of the initial segmentation labels, so that the energy function is iteratively updated to optimize the target segmentation labels.

7. An electronic device, characterized in that: The device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the target motion estimation method as described in any one of claims 1 to 6.

8. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by the processor, it is used to implement the target motion estimation method according to any one of claims 1 to 6.

Images