Method and device for determining scene flow, storage medium, and electronic device
By constructing and optimizing the scene flow energy function and calculating the scene flow using the LSTM network, the problem of insufficient speed and accuracy in the existing technology is solved, and more efficient scene flow calculation is achieved.
Patent Information
- Application Number
- CN202011174348.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-28
- Publication Date
- 2025-01-10
- Estimated Expiration
- 2040-10-28
AI Technical Summary
In the prior art, the energy function speed and accuracy of calculating scene flow are insufficient, especially in deep learning methods, where the solution is limited due to the lack of manual annotation data.
By obtaining the instance segmentation results, optical flow results and parallax results of the binocular image, the scene flow energy function is constructed, and the long-term short-term memory LSTM network is used to optimize the energy function to determine the target scene flow.
The solution speed and accuracy of scene flow energy functions are improved, the problem of lack of labeled data in deep learning methods is overcome, and more efficient scene flow calculation is achieved.
Smart Images
Figure CN112233149B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of images, and in particular, to a method and device for determining a scene flow, a storage medium, and an electronic device. Background Art
[0002] Scene flow estimation aims to estimate the three-dimensional motion vector through two consecutive binocular images, that is, the change of each point in the image between the previous and next frames. Accurate estimation of scene flow is extremely important for the development of robotics, in which it is used to achieve autonomous navigation and manipulation in dynamic environments. Estimation of scene flow not only allows the machine to accurately locate, but also improves the perception and prediction of the movement of surrounding objects, thereby avoiding collisions in the process of planning paths. At present, there are two main methods for calculating scene flow, namely, based on binocular stereo vision and based on red, green and blue depth map RGBD. The use of depth sensors can directly use sensors to obtain more accurate depth information, which can improve accuracy and save time. However, the RGBD-based method is prone to errors or even mistakes in occlusion. The method based on binocular stereo matching mainly infers the disparity and depth information from the left and right images taken by the binocular camera, as well as the optical flow information of the previous and next two frames of images, and solves the scene flow results through the depth information and optical flow information. This method consumes a lot of extra time because it estimates the disparity map, and the process of solving the scene flow through optical flow and disparity information is very time-consuming, and it is difficult to achieve real-time prediction. In addition, deep learning has gradually surpassed traditional algorithms in various fields. However, deep learning is heavily dependent on the accuracy of manually labeled data, and the scene flow results represent the motion state of each pixel, which cannot be labeled, thus limiting the use of deep learning methods to solve problems.
[0003] With regard to the problem of insufficient speed and accuracy in solving energy functions in the prior art, no effective solution has been proposed in the related art. Summary of the invention
[0004] The embodiments of the present invention provide a method and device for determining a scene flow, a storage medium, and an electronic device, so as to at least solve the problem of insufficient speed and accuracy in solving an energy function in the related art.
[0005] According to one embodiment of the present invention, a method for determining a scene flow is provided, comprising: determining processing parameters of a pixel point in an acquired binocular image, wherein the processing parameters include an instance segmentation result, an optical flow result, and a disparity result of the binocular image; determining a scene flow energy function of the binocular image based on the processing parameters; optimizing the scene flow energy function to obtain a target scene flow energy function; and determining the scene flow of the pixel point in the binocular image based on the target scene flow energy function.
[0006] According to another embodiment of the present invention, a scene flow determination device is provided, comprising: a first determination module, used to determine processing parameters of pixel points in an acquired binocular image, wherein the processing parameters include instance segmentation results, optical flow results and disparity results of the binocular image; a second determination module, used to determine a scene flow energy function of the binocular image based on the processing parameters; a third determination module, used to optimize the scene flow energy function to obtain a target scene flow energy function; and a fourth determination module, used to determine the scene flow of the pixel points in the binocular image based on the target scene flow energy function.
[0007] In an exemplary embodiment, the first determination module includes: a first acquisition unit, used to acquire the binocular image using a binocular camera device, wherein the binocular image includes an N-th left image and an N-th right image, wherein N is a natural number greater than or equal to 1; a first determination unit, used to perform instance segmentation on the binocular image through a preset instance segmentation network to obtain the instance segmentation result; a second determination unit, used to determine the displacement of the pixel points in the binocular image in a preset coordinate system through a preset optical flow estimation network to obtain the optical flow result; a third determination unit, used to determine the coordinate offset of the pixel points in the binocular image in the preset coordinate system through a preset stereo matching network to obtain the disparity result; and a fourth determination unit, used to determine the segmentation result, the optical flow result and the disparity result as the processing parameters.
[0008] In an exemplary embodiment, the second determination module includes: a fifth determination unit, used to determine the photometric error constraint, rigid fitting constraint and optical flow consistency constraint of the binocular image using the instance segmentation result, optical flow result, disparity result and rotation and translation matrix, wherein the rotation and translation matrix is determined based on the rotation and translation amount of the target object in the binocular image in adjacent frame images; a sixth determination unit, used to determine the scene flow energy function of the binocular image using the photometric error constraint, rigid fitting constraint and optical flow consistency constraint.
[0009] In an exemplary embodiment, the sixth determining unit includes: a first determining subunit, configured to determine E i =E photo,i +E rigid,i +E flow,i ; Among them, the above E i Used to represent the above scene flow energy function, the above E photo,i It is used to represent the above photometric error constraint term, E rigid,i It is used to represent the above rigid fitting constraint terms and E flow,i It is used to represent the above optical flow consistency constraint item, and the above i is an integer greater than or equal to 0.
[0010] In an exemplary embodiment, the fifth determining unit includes: a second determining subunit for determining Among them, the above E photo,i It is used to represent the above photometric error constraint term, E rigid,i It is used to represent the above rigid fitting constraint terms and E flow,i It is used to represent the above optical flow consistency constraint term, the above i is an integer greater than or equal to 0, the above α p It is used to indicate whether the current pixel in the above binocular image is an indicator function, and the above p is used to indicate the current instance P in the above binocular image. i The pixel points in the above P i Used to represent the left image L in the above binocular image 0 A collection of instances of In the i-th instance, the above p' is used to represent the pixel point of the three-dimensional point cloud of the above instance in the Nth frame p after being reprojected to the N+1th frame by the internal and external parameters of the camera device, and RT is used to represent the above rotation and translation matrix.
[0011] In an exemplary embodiment, the above p' is determined by: Among them, the above π K It is used to indicate that a three-dimensional point cloud is projected onto the imaging plane of the above-mentioned camera device using the intrinsic parameters of the camera device to obtain a two-dimensional image; the above-mentioned Used to represent the three-dimensional point cloud reconstructed by the stereo calibration parameters of the above-mentioned camera device using the two-dimensional coordinates and the disparity map; the above-mentioned q is used to represent the pixel point p in the above-mentioned binocular image plus the pixel point after the optical flow value corresponding to the above-mentioned p is added.
[0012] In an exemplary embodiment, the third determination module includes: a seventh determination unit, which is used to optimize the scene flow energy function using a long short-term memory (LSTM) network to obtain the target scene flow energy function, wherein the LSTM network includes a hidden layer LSTM and a linear layer.
[0013] In an exemplary embodiment, the seventh determination unit includes: a third determination subunit, used to determine the number of training times and the number of expansion times, wherein the number of training times is used to indicate the number of times the LSTM network is updated, and the number of expansion times is used to indicate the number of times the rotation and translation matrix is updated; a fourth determination subunit, used to initialize the LSTM network to obtain an initial LSTM network; and a fifth determination subunit, used to input the read sampling processing parameters and the initial rotation and translation matrix into the initial LSTM network when the number of times the LSTM network is updated is less than the number of training times and the number of times the rotation and translation matrix is updated is less than the number of expansion times, so as to train the initial LSTM network and obtain the target scene flow energy function.
[0014] In an exemplary embodiment, the above-mentioned device also includes: a testing module, which is used to test the above-mentioned target scene flow energy function using the processing parameters of the pixel points in the above-mentioned binocular image and the initial rotation and translation matrix RT after optimizing the above-mentioned scene flow energy function to obtain the target scene flow energy function.
[0015] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.
[0016] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0017] Through the present invention, by determining the processing parameters of the pixel points in the acquired binocular image, wherein the processing parameters include the instance segmentation result, optical flow result and disparity result of the binocular image; determining the scene flow energy function of the binocular image based on the processing parameters; optimizing the scene flow energy function to obtain the target scene flow energy function; and determining the scene flow of the pixel points in the binocular image based on the target scene flow energy function. It is achieved that the target scene flow function is determined using a relatively small number of parameters. Therefore, the problem of insufficient speed and accuracy in solving the energy function in the related art can be solved, and the effect of improving the speed and accuracy of solving complex energy functions can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a hardware structure block diagram of a mobile terminal of a method for determining a scene flow according to an embodiment of the present invention;
[0019] Figure 2 is a flowchart of a method for determining a scene flow according to an embodiment of the present invention;
[0020] Figure 3 is an overall flow chart according to an embodiment of the present invention;
[0021] Figure 4 is a flow chart of the implementation process of training according to an embodiment of the present invention;
[0022] Figure 5 is a flowchart of predicting an LSTM optimization model according to an embodiment of the present invention;
[0023] Figure 6 is a schematic diagram of a comparison result according to an embodiment of the present invention;
[0024] Figure 74 is a structural block diagram of a device for determining a scene flow according to an embodiment of the present invention. DETAILED DESCRIPTION
[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.
[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a method for determining a scene flow according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the scene flow in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0030] In this embodiment, a method for determining a scene flow is provided. Figure 2 is a flow chart of a method for determining a scene flow according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0031] Step S202, determining processing parameters of the pixels in the acquired binocular image, wherein the processing parameters include instance segmentation results, optical flow results, and disparity results of the binocular image;
[0032] Step S204, determining a scene flow energy function of the binocular image based on the processing parameters;
[0033] Step S206, optimizing the scene flow energy function to obtain a target scene flow energy function;
[0034] Step S208: determining the scene flow of the pixel points in the binocular image based on the target scene flow energy function.
[0035] The execution subject of the above steps may be a terminal, etc., but is not limited thereto.
[0036] Through the above steps, the processing parameters of the pixels in the acquired binocular image are determined, wherein the processing parameters include the instance segmentation result, optical flow result and disparity result of the binocular image; the scene flow energy function of the binocular image is determined based on the processing parameters; the scene flow energy function is optimized to obtain the target scene flow energy function; the scene flow of the pixels in the binocular image is determined based on the target scene flow energy function. The target scene flow function is determined using a relatively small number of parameters. Therefore, the problem of insufficient speed and accuracy in solving the energy function in the related art can be solved, and the speed and accuracy of solving complex energy functions can be improved.
[0037] In an exemplary embodiment, determining processing parameters of pixels in an acquired binocular image includes:
[0038] S1, using a binocular camera device to acquire a binocular image, wherein the binocular image includes an N-th left image and an N-th right image, wherein N is a natural number greater than or equal to 1;
[0039] S2, performing instance segmentation on the binocular image through a preset instance segmentation network to obtain an instance segmentation result;
[0040] S3, determining the displacement of the pixel points in the binocular image in the preset coordinate system through a preset optical flow estimation network to obtain an optical flow result;
[0041] S4, determining the coordinate offset of the pixel points in the binocular image in the preset coordinate system through a preset stereo matching network to obtain a disparity result;
[0042] S5, determining the segmentation result, the optical flow result and the disparity result as processing parameters.
[0043] Optionally, in this embodiment, the left and right eye images of consecutive frames can be obtained by a binocular camera. For example, the left and right eye images are subjected to stereo calibration and correction to obtain the internal and external parameters of the camera and the corrected image. The first left image and the second left image can obtain the displacement of each pixel of the first left image in the x and y directions through the optical flow network, that is, the optical flow result; the first left image can obtain the offset of each pixel of the first left image in the x direction through the binocular stereo matching network, that is, the disparity result; the first left image can obtain the instance segmentation result through the instance segmentation network.
[0044] In an exemplary embodiment, determining a scene flow energy function of a binocular image based on processing parameters includes:
[0045] S1, determining the photometric error constraint item, the rigid fitting constraint item and the optical flow consistency constraint item of the binocular image by using the instance segmentation result, the optical flow result, the disparity result and the rotation and translation matrix, wherein the rotation and translation matrix is determined based on the rotation and translation amount of the target object in the binocular image in adjacent frame images;
[0046] S2, determine the scene flow energy function of the binocular image using the photometric error constraint, the rigid fitting constraint, and the optical flow consistency constraint.
[0047] In an exemplary embodiment, the scene flow energy function of the binocular image is determined using a photometric error constraint, a rigid fitting constraint, and an optical flow consistency constraint, including:
[0048] E i =E photo,i +E rigid,i +E flow,i ;
[0049] Among them, E i Used to represent the scene flow energy function, E photo,i Used to represent the photometric error constraint term, E rigid,i Used to represent rigid fitting constraints and Eflow,i Used to represent the optical flow consistency constraint, i is an integer greater than or equal to 0.
[0050] Optionally, in this embodiment, for example, L 0 ,R 0 ,L 1 ,R 1 Represents the left and right images of two consecutive binocular images, respectively. 0 ,D 1 Represent the disparity map of frame 0 and frame 1 respectively, F L ,F R They represent the optical flow graphs corresponding to two consecutive frames of the left and right eyes respectively. represents the instance segmentation result of the 0th frame of the left eye image, and RT represents the rotation and translation matrix of the target between the previous and next frames. The construction of the scene flow in this embodiment includes three parts, which respectively construct the scene flow energy function from the gray value, 3D point cloud and optical flow of each instance in the scene based on the photometric error constraint, the rigid fitting constraint and the optical flow consistency constraint.
[0051] In an exemplary embodiment, the photometric error constraint term, the rigid fitting constraint term and the optical flow consistency constraint term of the binocular image are determined using the instance segmentation result, the optical flow result, the disparity result and the rotation and translation matrix, including:
[0052]
[0053]
[0054]
[0055] Among them, E photo,i It is used to represent the photometric error constraint term, E rigid,i Used to represent rigid fitting constraints and E flow,i It is used to represent the optical flow consistency constraint term, i is an integer greater than or equal to 0, α p It is used to indicate whether the current pixel in the binocular image is an indicator function, and p is used to indicate the current instance P in the binocular image. i The pixel point in i Used to represent the left image L in the binocular image 0 The instance set S L 0 In the i-th instance, p' is used to represent the pixel point of the 3D point cloud of the instance in the Nth frame p after being reprojected to the N+1th frame by the internal and external parameters of the camera device, and RT is used to represent the rotation and translation matrix.
[0056] In one exemplary embodiment, p' is determined by:
[0057]
[0058] Among them, π K It is used to indicate that a three-dimensional point cloud is projected onto the imaging plane of the camera device using the intrinsic parameters of the camera device to obtain a two-dimensional image; It is used to represent the three-dimensional point cloud reconstructed by the stereo calibration parameters of the camera device using the two-dimensional coordinates and the disparity map; q is used to represent the pixel point p in the binocular image plus the optical flow value corresponding to p.
[0059] In an exemplary embodiment, optimizing a scene flow energy function to obtain a target scene flow energy function includes:
[0060] S1, optimize the scene flow energy function using the long short-term memory (LSTM) network to obtain the target scene flow energy function, where the LSTM network includes a hidden layer LSTM and a linear layer.
[0061] Optionally, in this embodiment, the LSTM network is used to simulate the gradient descent expansion for training and prediction, and the update of RT and the update of LSTM network are performed alternately during the training process.
[0062] In an exemplary embodiment, a long short-term memory (LSTM) network is used to optimize the scene flow energy function to obtain a target scene flow energy function, including:
[0063] S1, determine the number of training times and the number of expansion times, where the number of training times is used to indicate the number of times the LSTM network is updated, and the number of expansion times is used to indicate the number of times the rotation and translation matrices are updated;
[0064] S2, initialize the LSTM network to obtain the initial LSTM network;
[0065] S3, when the number of times the LSTM network is updated is less than the number of training times, and the number of times the rotation and translation matrices are updated is less than the number of expansion times, the read sampling processing parameters and the initial rotation and translation matrices are input into the initial LSTM network to train the initial LSTM network and obtain the target scene flow energy function.
[0066] Optionally, in this embodiment, the LSTM is randomly initialized by setting the number of training times and the number of expansion times. Training is performed based on the number of training times and the number of expansion times.
[0067] In an exemplary embodiment, after optimizing the scene flow energy function and obtaining the target scene flow energy function, the method further includes:
[0068] S1, using the processing parameters of the pixels in the binocular image and the initial rotation and translation matrix RT to test the target scene flow energy function.
[0069] Optionally, in this embodiment, a group of left and right images of consecutive frames are obtained, and the initial RT and energy function expression can be obtained through instance segmentation, disparity estimation, and optical flow estimation. In this embodiment, optimization is performed separately in units of instances. For each instance, the number of LSTM expansions n is given, and the current loss value is obtained according to the RT input energy function, and the gradient of loss to RT is calculated, which is input into the LSTM to obtain the RT update amount, and the RT is updated; until after updating n times, the RT value corresponding to the minimum loss value is output, which is the motion state of the instance.
[0070] The present invention will be described below in conjunction with specific embodiments:
[0071] In this embodiment, the scene flow is used to represent the three-dimensional motion vector of each pixel in the image in the real scene, which is represented by a set of RT. The scene flow can be calculated from two consecutive frames of binocular images. Generally, the optical flow is calculated from the two frames before and after. The optical flow represents the movement displacement of the pixel in the x and y directions. The disparity can be calculated based on the left and right images of the binoculars, and the depth of the point can be estimated based on the disparity and the internal and external parameters of the camera. The scene flow energy function is constructed based on the optical flow results and the disparity results, and the scene flow results can be obtained after optimization.
[0072] like Figure 3 FIG. 4 is the overall process of this embodiment, which includes the following steps:
[0073] S301: Continuous frames of left and right eye images can be obtained through a binocular camera. After the left and right eye images are stereo calibrated and corrected, the camera internal and external parameters and the corrected image are obtained.
[0074] S302: The first left frame image and the second left frame image can obtain the displacement of each pixel in the first left frame image in the x and y directions through the optical flow network; the first left and right frames image can obtain the offset of each pixel in the first left frame image in the x direction, i.e., the disparity, through the binocular stereo matching network; the first left frame image can obtain the instance segmentation result through the instance segmentation network.
[0075] S303: Construct a scene flow energy function through instance segmentation results, optical flow results, and disparity results, and calculate an initial RT.
[0076] S304: Use LSTM to gradually optimize the energy function, continuously update the RT, and finally obtain the RT of each instance.
[0077] S305: Obtain scene flow results.
[0078] Optionally, the scene flow energy function is constructed as follows:
[0079] This embodiment uses the deep learning stereo matching network Dispnet to perform disparity estimation, the deep learning optical flow estimation network FlowNet to perform optical flow calculation, and the Centermask network to perform instance segmentation result calculation. The instance-based scene flow estimation treats each instance as a whole and estimates the same motion vector, so the instance segmentation module only segments various vehicles and backgrounds.
[0080] L 0 ,R 0 ,L 1 ,R 1 They represent the left and right images of two consecutive binocular frames, respectively. 0 ,D 1 Represent the disparity map of frame 0 and frame 1 respectively, F L ,F R They represent the optical flow graphs corresponding to two consecutive frames of the left and right eyes respectively. represents the instance segmentation result of the 0th frame of the left image, and RT represents the rotation and translation matrix of the target between the previous and next frames. The construction of the scene flow in this proposal consists of three parts. Based on the photometric error constraint, rigid fitting constraint, and optical flow consistency constraint, the energy function shown in Formula 1 is constructed from the grayscale value, 3D point cloud, and optical flow of each instance in the scene.
[0081] E i =E photo,i +E rigid,i +E flow,i Formula 1;
[0082]
[0083]
[0084]
[0085] The photometric error constraint, rigid fitting constraint, and optical flow consistency constraint are shown in formulas 2-4 respectively.
[0086] Among them, α p Is an indicator function indicating whether the current pixel is an outlier. A value of 0 indicates an outlier. p represents the current instance P i The pixel point in i Indicates L 0 Instance Collection p' represents the pixel point of the 3D point cloud of p in frame 0 after being reprojected to frame 1 by the camera's internal and external parameters. The transformation relationship can be expressed by the following formula:
[0087]
[0088] Among them, π K It means that the three-dimensional point cloud is projected onto the camera imaging plane using the camera intrinsic parameters to obtain a two-dimensional image. Indicates the reconstruction of a 3D point cloud using 2D coordinates and disparity maps through camera stereo calibration parameters. q is the pixel point p plus its corresponding optical flow value, q = p + F L (p).
[0089]
[0090] Optionally, the optimization of the LSTM-based scene flow energy function includes the following:
[0091] The LSTM network in this embodiment is an LSTM layer including two hidden layers and one linear layer. The input of the network is the gradient of the energy function to RT, and the output is the update amount of RT. The input and output sizes are both (B, 6).
[0092] The LSTM network in this embodiment is a meta-optimizer, which has the same function as the traditional optimizers Gauss-Newton method, Adam, and RMS, and predicts the update amount based on the gradient. The difference is that the LSTM network is data-dependent and trainable.
[0093] Optionally, in the training set, for two consecutive binocular images, the semantic segmentation results, disparity results and optical flow results are calculated using the pre-trained semantic segmentation network, disparity estimation network and optical flow estimation network, respectively. Based on these results, the initial RT and the corresponding energy function corresponding to each instance can be calculated.
[0094] This embodiment uses the LSTM network to simulate the gradient descent expansion for training and prediction. The RT update and the LSTM network update are performed alternately during the training process. Figure 4 As shown, the following steps are included:
[0095] S401: Set the number of training times and the number of expansion times, and randomly initialize the LSTM. The number of training times is the number of times the LSTM network is updated according to the training set, and the number of expansion times is the number of times the RT is updated during each training process.
[0096] S402: Determine whether it is less than the number of training times. If yes, go to S403. If not, output the model (S408) and the training ends. Read a batch of training data, including two consecutive frames of binocular images, semantic segmentation results, optical flow and disparity results, and initial RT.
[0097] S403: Determine whether it is less than the expansion times, if not, go to S410; if yes, go to S404;
[0098] S404-S405, S409: Calculate the energy function loss according to RT, calculate the gradient of loss with respect to RT, input the RT update amount △RT corresponding to the LSTM prediction, update RT, RT=RT+△RT.
[0099] S406: sum each loss in the previous RT process to obtain sum_loss, and update the LSTM network (S410);
[0100] S407: Update RT, and then repeat S402-S410.
[0101] Optionally, this embodiment predicts the LSTM optimization model in the following manner: Figure 5 As shown, the following steps are included:
[0102] S501: Given a set of left and right images of consecutive frames, the initial RT and energy function expression can be obtained through instance segmentation, disparity estimation, and optical flow estimation.
[0103] S502: In this embodiment, optimization is performed on an instance-by-instance basis. For each instance, a given LSTM expansion number n is given. If the number is less than the expansion number, the process goes to S503; otherwise, the process goes to S508.
[0104] S503: Obtain the current loss value according to the RT input energy function, and go to S507 (energy function collection);
[0105] S504: Calculate the gradient of loss with respect to RT;
[0106] S505: Input into LSTM to obtain the update amount of RT:
[0107] S506: Update RT, and go to S509 (RT collection);
[0108] S508: After updating n times, the RT value corresponding to the minimum loss value is output, which is the motion state of the instance.
[0109] Optionally, in this embodiment, the LSTM optimization model effect includes: taking the KITTI dataset as an example, three optimizers, LSTM, RMS, and Adam, are compared. The comparison results are as follows: Figure 6 As shown. Figure 6 It can be seen that LSTM greatly surpasses the traditional RMS and Adam optimizers in terms of convergence speed and accuracy.
[0110] In summary, the LSTM optimization modules in this embodiment are all trainable. After training on a specific data set, they can fit the data set well, so that higher scene flow accuracy can be obtained. Compared with traditional optimization methods such as Gauss-Newton method, Adam, RMS, etc., which require a lot of calculations to obtain parameter updates, this proposal adopts only two hidden layers of LSTM layer and linear layer structure, which has small amount of calculation and low calculation efficiency, and provides great advantages for the implementation of scene flow algorithm. The LSTM optimization network in this embodiment can be trained without the true value of the scene flow, which solves the problem that the scene flow results cannot be labeled, and also provides a new idea for solving the scene flow accuracy using deep learning methods. The LSTM optimization network in this embodiment is a problem-independent meta-learner that can be applied to the solution of any objective function and has very strong scalability.
[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0112] In this embodiment, a scene flow determination device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0113] Figure 7 is a structural block diagram of a device for determining a scene flow according to an embodiment of the present invention, such as Figure 7 As shown, the device comprises:
[0114] A first determination module 72, used to determine processing parameters of pixels in the acquired binocular image, wherein the processing parameters include instance segmentation results, optical flow results, and disparity results of the binocular image;
[0115] A second determination module 74, configured to determine a scene flow energy function of the binocular image based on the processing parameters;
[0116] A third determination module 76 is used to optimize the scene flow energy function to obtain a target scene flow energy function;
[0117] The fourth determination module 78 is used to determine the scene flow of the pixel points in the binocular image based on the target scene flow energy function.
[0118] In an exemplary embodiment, the first determining module includes:
[0119] A first acquisition unit is used to acquire the binocular image using a binocular camera device, wherein the binocular image includes an N-th left image and an N-th right image, wherein N is a natural number greater than or equal to 1;
[0120] A first determining unit is used to perform instance segmentation on the binocular image through a preset instance segmentation network to obtain the instance segmentation result;
[0121] A second determining unit is used to determine the displacement of the pixel points in the above binocular image in a preset coordinate system through a preset optical flow estimation network to obtain the above optical flow result;
[0122] A third determining unit is used to determine the coordinate offset of the pixel points in the binocular image in the preset coordinate system through a preset stereo matching network to obtain the disparity result;
[0123] The fourth determining unit is used to determine the segmentation result, the optical flow result and the disparity result as the processing parameters.
[0124] In an exemplary embodiment, the second determining module includes:
[0125] A fifth determining unit is used to determine the photometric error constraint item, the rigid fitting constraint item and the optical flow consistency constraint item of the above-mentioned binocular image by using the above-mentioned instance segmentation result, the optical flow result, the disparity result and the rotation and translation matrix, wherein the above-mentioned rotation and translation matrix is determined based on the rotation and translation amount of the target object in the above-mentioned binocular image in adjacent frame images;
[0126] The sixth determining unit is used to determine the scene flow energy function of the binocular image by using the photometric error constraint, the rigid fitting constraint and the optical flow consistency constraint.
[0127] In an exemplary embodiment, the sixth determining unit includes:
[0128] The first determining subunit is used to determine E i =E photo,i +E rigid,i +E flow,i ; Among them, the above E i Used to represent the above scene flow energy function, the above E photo,i It is used to represent the above photometric error constraint term, E rigid,iIt is used to represent the above rigid fitting constraint terms and E flow,i It is used to represent the above optical flow consistency constraint item, and the above i is an integer greater than or equal to 0.
[0129] In an exemplary embodiment, the fifth determining unit includes: a second determining subunit for determining Among them, the above E photo,i It is used to represent the above photometric error constraint term, E rigid,i It is used to represent the above rigid fitting constraint terms and E flow,i It is used to represent the above optical flow consistency constraint term, the above i is an integer greater than or equal to 0, the above α p It is used to indicate whether the current pixel in the above binocular image is an indicator function, and the above p is used to indicate the current instance P in the above binocular image. i The pixel points in the above P i Used to represent the left image L in the above binocular image 0 A collection of instances of In the i-th instance, the above p' is used to represent the pixel point of the three-dimensional point cloud of the above instance in the Nth frame p after being reprojected to the N+1th frame by the internal and external parameters of the camera device, and RT is used to represent the above rotation and translation matrix.
[0130] In an exemplary embodiment, the above p' is determined by: Among them, the above π K It is used to indicate that a three-dimensional point cloud is projected onto the imaging plane of the above-mentioned camera device using the intrinsic parameters of the camera device to obtain a two-dimensional image; the above-mentioned Used to represent the three-dimensional point cloud reconstructed by the stereo calibration parameters of the above-mentioned camera device using the two-dimensional coordinates and the disparity map; the above-mentioned q is used to represent the pixel point p in the above-mentioned binocular image plus the pixel point after the optical flow value corresponding to the above-mentioned p is added.
[0131] In an exemplary embodiment, the third determining module includes:
[0132] The seventh determination unit is used to optimize the scene flow energy function by using a long short-term memory (LSTM) network to obtain the target scene flow energy function, wherein the LSTM network includes a hidden layer LSTM and a linear layer.
[0133] In an exemplary embodiment, the seventh determining unit includes:
[0134] A third determination subunit is used to determine the number of training times and the number of expansion times, wherein the number of training times is used to indicate the number of times the LSTM network is updated, and the number of expansion times is used to indicate the number of times the rotation and translation matrix is updated;
[0135] The fourth determination subunit is used to initialize the above LSTM network to obtain an initial LSTM network;
[0136] The fifth determination subunit is used to input the read sampling processing parameters and the initial rotation and translation matrix into the above-mentioned initial LSTM network when the number of times the above-mentioned LSTM network is updated is less than the above-mentioned training number and the number of times the rotation and translation matrix is updated is less than the above-mentioned expansion number, so as to train the above-mentioned initial LSTM network and obtain the above-mentioned target scene flow energy function.
[0137] In an exemplary embodiment, the above-mentioned device also includes: a testing module, which is used to test the above-mentioned target scene flow energy function using the processing parameters of the pixel points in the above-mentioned binocular image and the initial rotation and translation matrix RT after optimizing the above-mentioned scene flow energy function to obtain the target scene flow energy function.
[0138] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0139] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.
[0140] In this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:
[0141] S1, determining processing parameters of pixels in the acquired binocular image, wherein the processing parameters include instance segmentation results, optical flow results, and disparity results of the binocular image;
[0142] S2, determining the scene flow energy function of the binocular image based on the processing parameters;
[0143] S3, optimize the scene flow energy function to obtain the target scene flow energy function;
[0144] S4, determining the scene flow of the pixel points in the binocular image based on the target scene flow energy function.
[0145] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0146] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0147] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0148] In an exemplary embodiment, the processor may be configured to perform the following steps via a computer program:
[0149] S1, determining processing parameters of pixels in the acquired binocular image, wherein the processing parameters include instance segmentation results, optical flow results, and disparity results of the binocular image;
[0150] S2, determining the scene flow energy function of the binocular image based on the processing parameters;
[0151] S3, optimize the scene flow energy function to obtain the target scene flow energy function;
[0152] S4, determining the scene flow of the pixel points in the binocular image based on the target scene flow energy function.
[0153] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0154] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for determining a scene flow, characterized in that: include: Determining processing parameters of pixels in the acquired binocular image, wherein the processing parameters include instance segmentation results, optical flow results, and disparity results of the binocular image; Determining a scene flow energy function of the binocular image based on the processing parameters; Optimizing the scene flow energy function to obtain a target scene flow energy function; Determining the scene flow of the pixel points in the binocular image based on the target scene flow energy function; The step of optimizing the scene flow energy function to obtain a target scene flow energy function comprises: optimizing the scene flow energy function using a long short-term memory (LSTM) network to obtain the target scene flow energy function, wherein the LSTM network comprises a hidden layer LSTM and a linear layer; Optimizing the scene flow energy function using a long short-term memory (LSTM) network to obtain the target scene flow energy function includes: determining a training number and an expansion number, wherein the training number is used to indicate the number of times the LSTM network is updated, and the expansion number is used to indicate the number of times the rotation and translation matrix is updated; initializing the LSTM network to obtain an initial LSTM network; when the number of times the LSTM network is updated is less than the training number, and the number of times the rotation and translation matrix is updated is less than the expansion number, inputting the read sampling processing parameters and the initial rotation and translation matrix into the initial LSTM network to train the initial LSTM network to obtain the target scene flow energy function; Wherein, determining the scene flow energy function of the binocular image based on the processing parameters includes: using the instance segmentation result, the optical flow result, the disparity result and the rotation and translation matrix to determine the photometric error constraint item, the rigid fitting constraint item and the optical flow consistency constraint item of the binocular image, wherein the rotation and translation matrix is determined based on the rotation and translation amount of the target object in the binocular image in adjacent frame images; and using the photometric error constraint item, the rigid fitting constraint item and the optical flow consistency constraint item to determine the scene flow energy function of the binocular image.
2. The method according to claim 1, characterized in that Determine the processing parameters of the pixels in the acquired binocular image, including: Acquire the binocular image using a binocular camera device, wherein the binocular image includes an N-th left image and an N-th right image, wherein N is a natural number greater than or equal to 1; Performing instance segmentation on the binocular image through a preset instance segmentation network to obtain the instance segmentation result; Determine the displacement of the pixel points in the binocular image in a preset coordinate system through a preset optical flow estimation network to obtain the optical flow result; Determine the coordinate offset of the pixel points in the binocular image in the preset coordinate system through a preset stereo matching network to obtain the disparity result; The segmentation result, the optical flow result, and the disparity result are determined as the processing parameters.
3. The method according to claim 1, characterized in that Determining the scene flow energy function of the binocular image by using the photometric error constraint, the rigid fitting constraint, and the optical flow consistency constraint includes: AND i =And photo,i +E rigid,i +E flow,i ; Among them, the E i It is used to represent the scene flow energy function, the E photo,i It is used to represent the photometric error constraint term, E rigid,i is used to represent the rigid fitting constraint term and E flow,i Used to represent the optical flow consistency constraint item, and i is an integer greater than or equal to 0.
4. The method according to claim 1, characterized in that: Determining the photometric error constraint item, the rigid fitting constraint item and the optical flow consistency constraint item of the binocular image by using the instance segmentation result, the optical flow result, the disparity result and the rotation and translation matrix, including: Among them, the E photo,i It is used to represent the photometric error constraint term, E rigid,i is used to represent the rigid fitting constraint term and E flow,i is used to represent the optical flow consistency constraint term, i is an integer greater than or equal to 0, and α p It is used to indicate whether the current pixel in the binocular image is an indicator function, and p is used to indicate the current instance P in the binocular image. i The pixel point in the i Used to represent the left image L in the binocular image 0 A collection of instances of In the i-th instance, the p' is used to represent the pixel point of the three-dimensional point cloud of the instance in the N-th frame p after being reprojected to the N+1-th frame by the internal and external parameters of the camera device, and RT is used to represent the rotation and translation matrix.
5. The method according to claim 4, characterized in that The p' is determined by: Among them, the π K It is used to indicate that a three-dimensional point cloud is projected onto an imaging plane of the camera device using the intrinsic parameters of the camera device to obtain a two-dimensional image; Used to represent the three-dimensional point cloud reconstructed by the stereo calibration parameters of the camera device using the two-dimensional coordinates and the disparity map; the q is used to represent the pixel point p in the binocular image plus the optical flow value corresponding to the p.
6. The method according to claim 1, characterized in that After optimizing the scene flow energy function to obtain a target scene flow energy function, the method further includes: The target scene flow energy function is tested using the processing parameters of the pixels in the binocular image and the initial rotation and translation matrix RT.
7. A device for determining scene flow, characterized in that: include: A first determination module is used to determine processing parameters of pixels in the acquired binocular image, wherein the processing parameters include instance segmentation results, optical flow results, and disparity results of the binocular image; A second determination module, configured to determine a scene flow energy function of the binocular image based on the processing parameters; A third determination module is used to optimize the scene flow energy function to obtain a target scene flow energy function; A fourth determination module, configured to determine a scene flow of a pixel point in the binocular image based on the target scene flow energy function; The third determination module is further used to optimize the scene flow energy function by using a long short-term memory (LSTM) network to obtain the target scene flow energy function, wherein the LSTM network includes a hidden layer LSTM and a linear layer; The third determination module is further used to determine the number of training times and the number of expansion times, wherein the number of training times is used to indicate the number of times the LSTM network is updated, and the number of expansion times is used to indicate the number of times the rotation and translation matrix is updated; initialize the LSTM network to obtain an initial LSTM network; when the number of times the LSTM network is updated is less than the number of training times, and the number of times the rotation and translation matrix is updated is less than the number of expansion times, input the read sampling processing parameters and the initial rotation and translation matrix into the initial LSTM network to train the initial LSTM network to obtain the target scene flow energy function; Among them, the second determination module is also used to determine the photometric error constraint item, rigid fitting constraint item and optical flow consistency constraint item of the binocular image by using the instance segmentation result, the optical flow result, the disparity result and the rotation and translation matrix, wherein the rotation and translation matrix is determined based on the rotation and translation amount of the target object in the binocular image in adjacent frame images; and the scene flow energy function of the binocular image is determined by using the photometric error constraint item, the rigid fitting constraint item and the optical flow consistency constraint item.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6 when executed.
9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.