Scene flow estimation device and method

Abstract

The present disclosure relates to a scene flow estimation apparatus, a scene flow estimation method, and a non-transitory computer-readable medium, the scene flow estimation apparatus including: a hierarchical feature detector that detects hierarchical features of point cloud data related to an image frame; the corrector corrects a hierarchical feature of the detected point cloud data, the bidirectional feature detector detects a bidirectional scene flow feature of the corrected point cloud data, re-inputs a detection value, and repeatedly detects the bidirectional scene flow feature, and the estimator estimates a scene flow of a next image frame by using the bidirectional scene flow feature.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2023-0086085 filed on July 3, 2023, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a scene flow estimation device and method. Background Art

[0004] Dual Point Flow Network refers to a deep neural network model that receives point cloud data of at least two consecutive frames as input and extracts scene flow features bidirectionally to predict the scene flow of the next frame. Here, point cloud data refers to data collected by LiDAR sensors, RGB-D sensors, etc., and means a set of points where the distance information of each beam of light / signal reflected on an object is generated as a point in 3D space.

[0005] The dual point flow network consists of four hierarchical structures and can be used to predict scene flow. The four hierarchical structures can include hierarchical feature extraction, bidirectional flow embedding, upsampling, and flow predictor.

[0006] The disadvantage of the two-point flow network is that it takes a long time to make accurate predictions due to the large amount of computation in the feature detection process and the flow embedding execution process at each of the source and target points.

[0007] In addition, the disadvantage of the two-point flow network is that the flow prediction performance deteriorates due to the application of hierarchical single-flow embedding layers. Summary of the invention

[0008] The present disclosure provides a scene flow estimation device and method, which apply a bidirectional flow embedding layer of an iterative execution structure to improve flow prediction accuracy.

[0009] Furthermore, the present disclosure provides a scene flow estimation apparatus and method, which apply an initial flow inference scheme using a voting function to reduce the number of calculation parameters and enhance flow prediction accuracy.

[0010] To achieve this purpose, an exemplary embodiment of the present disclosure provides a scene flow estimation device, which includes a controller, the controller includes a hierarchical feature detector, a corrector, a bidirectional feature detector and an inference device, the hierarchical feature detector detects hierarchical features of point cloud data related to an image frame, the corrector corrects the detected hierarchical features of the point cloud data, the bidirectional feature detector detects bidirectional scene flow features of the corrected point cloud data, re-inputs the detection value, and repeatedly detects the bidirectional scene flow features, and the inference device infers the scene flow of the next image frame by using the bidirectional scene flow features.

[0011] The bidirectional feature detector can detect the initial scene flow of point cloud data by using a predetermined voting block.

[0012] The directional feature detector calculates voting values ​​of group points of the grouping of point cloud data by applying SoftMax to the voting block, and adds the group points using the voting values ​​as weights to extract the initial moving positions for detecting the initial scene flow.

[0013] The inferer may infer the scene flow of the next image frame by correcting the initial scene flow using the bidirectional scene flow feature.

[0014] The corrector may apply an interpolation scheme to the hierarchical features of the detected point cloud data and upsample the hierarchical features.

[0015] The corrector may perform warping on the inference value of the inferencer and use the warped inference value as input for extracting hierarchical features of the point cloud data.

[0016] To achieve this objective, another exemplary embodiment of the present disclosure provides a scene flow estimation method, comprising: detecting, by a hierarchical feature detector of a controller, hierarchical features of point cloud data associated with an image frame; correcting, by a corrector of the controller, the detected hierarchical features of the point cloud data; detecting, by a bidirectional feature detector of the controller, bidirectional scene flow features of the corrected point cloud data, re-inputting the detection value, and repeatedly detecting the bidirectional scene flow features; and inferring, by an inferor of the controller, the scene flow of the next image frame by using the bidirectional scene flow features.

[0017] The scene flow estimation method may further include an initial scene detection step, wherein the bidirectional feature detector detects an initial scene flow of the point cloud data by using a predetermined voting block.

[0018] The initial scene detection step may include: calculating voting values ​​of group points of the grouping of point cloud data by applying SoftMax to the voting block by a bidirectional feature detector, and adding the group points using the voting values ​​as weights to extract an initial moving position for detecting the initial scene flow.

[0019] The inferring step may include inferring, by an inferer, the scene flow of the next image frame by correcting the initial scene flow using the bidirectional scene flow feature.

[0020] The correcting step may include applying, by a corrector, an interpolation scheme to the detected hierarchical features of the point cloud data and upsampling the hierarchical features.

[0021] The correcting step may include performing, by the corrector, warping on the inference value of the inference device, and using the warped inference value as an input for extracting the hierarchical features of the point cloud data.

[0022] Another exemplary embodiment of the present disclosure provides a non-transitory computer-readable medium containing program instructions executed by a processor, including: program instructions for detecting hierarchical features of point cloud data associated with an image frame; program instructions for correcting the detected hierarchical features of the point cloud data; program instructions for detecting bidirectional scene flow features of the corrected point cloud data, re-inputting detection values, and repeatedly detecting the bidirectional scene flow features; and program instructions for inferring a scene flow of a next image frame by using the bidirectional scene flow features.

[0023] According to an exemplary embodiment of the present disclosure, through a scene flow estimation apparatus, method, and non-transitory computer-readable medium, a bidirectional flow embedding layer is changed into an iterative execution structure so that the output is re-input to enhance the flow inference accuracy.

[0024] Furthermore, an initial flow is inferred by introducing a voting block for bidirectional features detected in each layer, and the flow predicted by the flow predictor is corrected by using the inferred initial flow to enhance the flow inference accuracy.

[0025] Furthermore, only the initial flows through voting instead of all flow-based flow inferences are used for flow inference to reduce the computation amount.

[0026] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a block diagram of a scene flow estimation apparatus according to an exemplary embodiment of the present disclosure.

[0028] Figure 2 is a diagram showing a hierarchical structure of a scene flow estimation apparatus according to an exemplary embodiment of the present disclosure.

[0029] Figure 3 is a diagram showing the iterative execution structure of the bidirectional stream embedding layer.

[0030] Figure 4A and Figure 4B It is shown Figure 3 Diagram of the source code for the iterative execution structure in .

[0031] Figure 5A and Figure 5B is a diagram illustrating an initial scene flow inference process of a scene flow estimation apparatus according to an exemplary embodiment of the present disclosure.

[0032] Fig. 6A and Figure 6Bis a diagram showing the source code of a voting function for initial scene flow inference.

[0033] Figure 7 The invention is a flowchart of an iterative execution process of a bidirectional flow embedding layer in a scene flow estimation method according to an exemplary embodiment of the present disclosure.

[0034] Figure 8 The invention is a flowchart including an initial scene flow inference process in a scene flow estimation method according to an exemplary embodiment of the present disclosure.

[0035] It should be understood that the accompanying drawings are not necessarily drawn to scale, and that they present somewhat simplified representations of various features illustrating the basic principles of the present disclosure. The specific design features of the present disclosure disclosed herein (including, for example, specific dimensions, orientations, locations, and shapes) will be determined in part by the specific intended application and use environment.

[0036] In the drawings, reference numbers refer to the same or equivalent parts of the present disclosure throughout the several figures of the drawing. DETAILED DESCRIPTION

[0037] The terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. As used herein, the singular forms "one", "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the terms "including" and / or "including" when used in this specification specify the existence of stated features, integral bodies, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, integral bodies, steps, operations, elements, parts and / or their groups. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Throughout the specification, unless explicitly described to the contrary, the word "including" and variations such as "including" or "including" will be understood to imply the inclusion of stated elements, but do not exclude any other elements. In addition, the terms "unit", "... piece", "... device" and "module" described in the specification are meant to be used to process a unit of at least one function and operation, and can be implemented by hardware components or software components and combinations thereof.

[0038] In addition, the control logic of the present disclosure may be implemented as a non-transitory computer-readable medium or computer-readable medium containing executable program instructions executed by a processor, a controller, etc. Examples of computer-readable media include, but are not limited to, ROM, RAM, compact disk (CD)-ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. The computer-readable medium may also be distributed in a network-connected computer system so that the executable program instructions are stored and executed in a distributed manner, for example, through a telematics server or a controller area network (CAN).

[0039] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. First, when a reference numeral represents a component of each figure, it should be noted that, although the same component is shown in different figures, the same component is represented by the same reference numeral as much as possible. In addition, hereinafter, exemplary embodiments of the present disclosure will be described, but the technical spirit of the present disclosure is not limited thereto or thereby, and the exemplary embodiments can be modified and performed differently by those skilled in the art.

[0040] Figure 1 is a block diagram of a scene flow estimation apparatus according to an exemplary embodiment of the present disclosure.

[0041] Reference Figure 1 According to an exemplary embodiment of the present disclosure, the scene flow estimation device 100, as a deep neural network model including a dual-point flow network, applies a bidirectional flow embedding layer of an iterative execution structure to enhance the flow prediction accuracy.

[0042] In addition, the scene flow estimation apparatus 100 according to the exemplary embodiment of the present disclosure applies an initial flow inference scheme using a voting function to reduce the number of calculation parameters and enhance flow prediction accuracy.

[0043] The scene flow estimation apparatus 100 according to an exemplary embodiment of the present disclosure includes a hierarchical feature detector 110 , a corrector 120 , a bidirectional feature detector 130 , and an inferencer 140 .

[0044] Each of the above elements may constitute a module and / or device of the scene flow estimation apparatus 100 that may serve as a controller. For example, the above elements of the scene flow estimation apparatus 100 may constitute a hardware component that forms a part of the controller (e.g., a module or device of a high-level controller), or may constitute a separate controller each having a processor and a memory. The scene flow estimation apparatus 100 may include one or more processors and memories.

[0045] The hierarchical feature detector 110 may be defined as a hierarchical feature extraction structure in four hierarchical structures of a double-point flow network.

[0046] The hierarchical feature detector 110 may receive point cloud data of at least two consecutive image frames. Here, the image frames may include a source image frame and a target image frame. The source image frame may be composed of a set of source points, and the target image frame may be composed of a set of target points.

[0047] The hierarchical feature detector 110 may be configured as PointConv and may detect hierarchical features of point cloud data. The hierarchical features may include staged features from a low step to a high step. The low step may have a low level and a high resolution. The high step may have a high level and a low resolution.

[0048] The corrector 120 may be defined as an up-sampling structure in a four-level structure of a dual-point flow network.

[0049] The corrector 120 may upsample the hierarchical features by applying an interpolation scheme to the hierarchical features of the detected point cloud data starting from a high stage, and propagate the upsampled high stage features and scene flow to a low stage.

[0050] The bidirectional feature detector 130 may be defined as a bidirectional flow embedding structure in four hierarchical structures of a dual-point flow network.

[0051] The bidirectional feature detector 130 may detect bidirectional scene flow features at each stage of the upsampled source points (SP) and target points (TP). The bidirectional feature detector 130 may repeatedly perform a bidirectional scene flow feature detection process by inputting a detection value.

[0052] Furthermore, the bidirectional feature detector 130 configures a voting block for each stage of the upsampled point cloud data to detect the initial scene flow.

[0053] The inferencer 140 may be defined as a flow predictor structure in a four-level structure of a two-point flow network.

[0054] The inferring unit 140 may infer the scene flow of the next image frame by using the bidirectional scene flow feature of each stage of the point cloud data. The inferring unit 140 may repeatedly perform the scene flow inference process of the next image frame by inputting the inference value.

[0055] Furthermore, the inferer 140 may infer the scene flow of the next image frame by correcting the initial scene flow using the bidirectional scene flow feature of each stage of the point cloud data.

[0056] Figure 2 is a diagram showing a hierarchical structure of a scene flow estimation apparatus according to an exemplary embodiment of the present disclosure.

[0057] Reference Figure 2 , point cloud data may include source points (SP) and target points (TP) of consecutive image frames.

[0058] The hierarchical feature detector 110 may detect a hierarchical feature of each of the source point SP and the target point TP in a hierarchical feature extraction structure.

[0059] The corrector 120 may upsample features by applying an interpolation scheme to the hierarchical features of the detected source points SP and target points TP in an upsampling (UP) structure starting from a high stage, and propagate the upsampled high stage features and scene flow to a low stage.

[0060] The bidirectional feature detector 130 may detect bidirectional scene flow features at each stage of up-sampled source points SP and target points TP in a bidirectional flow embedding (BFE) structure. The bidirectional feature detector 130 may repeatedly perform a bidirectional scene flow feature detection process by inputting a detection value.

[0061] Furthermore, the bidirectional feature detector 130 configures a voting block for each stage of upsampled source points SP and target points TP to detect the initial scene flow.

[0062] The inferer 140 may infer the scene flow V of the next image frame by using the bidirectional scene flow features of each stage of the source point SP and the target point TP in a flow predictor (FP) structure. The inferer 140 may repeatedly perform the scene flow inference process of the next image frame by inputting the inference value.

[0063] Furthermore, the inferer 140 may infer the scene flow V of the next image frame by correcting the initial scene flow using the bidirectional scene flow features of each stage of the point cloud data.

[0064] Figure 3 is a diagram showing the iterative execution structure of the bidirectional stream embedding layer.

[0065] Reference Figure 3 , we can see the iterative execution structure of the bidirectional stream embedding layer.

[0066] The bidirectional feature detector 130 may repeatedly apply a bidirectional flow embedding layer that detects features of bidirectional scene flow in order to improve prediction performance while maintaining the model size.

[0067] The bidirectional scene flow features detected in an iterative execution structure can be applied as input to the next iterative execution structure.

[0068] The flow predictor (FP) may correct the initial scene flow by using the bidirectional scene flow features detected in the iterative execution structure, and warp the target points and apply the warped target points as the input of the next iterative execution structure.

[0069] The iterative execution structure can be applied to each layer of the bidirectional stream embedding.

[0070] Hereinafter, the bidirectional feature detection process of the bidirectional stream embedding layer applying the iterative execution structure will be described in detail.

[0071] The bidirectional feature detector 130 performs k-nearest neighbors (kNN) on all points of the source frame and the target frame in an iterative execution structure to collect neighboring point groups from the opposite frames.

[0072] The bidirectional feature detector 130 combines the feature of a point in each of the source frame and the target frame with the features of a neighboring point group to update the feature of the point.

[0073] The bidirectional feature detector 130 inputs the updated point features into the next iteration execution structure according to the virtual input stream IF to repeatedly update the point features.

[0074] Since the bidirectional feature detector 130 is based on a dual point flow network structure, all updates can be completed in the previous stage, and then the point features can be passed to the upsampling layer of the next stage. The point features can be updated according to the iterative execution structure of the next stage.

[0075] The point feature update according to the iterative execution structure may be performed a predetermined number of repeated executions.

[0076] Updating the content of a point feature using a relative feature in an iterative execution structure can be defined by repeated bidirectional feature detection.

[0077] The repeatedly updated point features can be used to infer scene flow.

[0078] Figure 4A and Figure 4B It is shown Figure 3 Diagram of the source code for the iterative execution structure in .

[0079] Reference Figure 4A and Figure 4B , you can see the source code of the bidirectional feature detector 130.

[0080] Figure 4A The forward path source code of the bidirectional feature detector 130 is shown before applying the iterative execution structure.

[0081] Figure 4B The forward path source code of the bidirectional feature detector 130 to which the iterative execution structure is applied is shown.

[0082] Figure 5A and Figure 5B is a diagram illustrating an initial scene flow inference process of a scene flow estimation apparatus according to an exemplary embodiment of the present disclosure.

[0083] Figure 5A is a diagram showing a process for extracting the initial movement position of the source point SP.

[0084] Figure 5B is a diagram showing the initial scene flow inference process.

[0085] Reference Figure 5A and Figure 5B , the bidirectional feature detector 130 configures a voting block for each stage of the upsampled source points SP and target points TP to detect the initial scene flow.

[0086] The bidirectional feature detector 130 introduces SoftMax in the bidirectional flow propagation (BFP) block of the bidirectional flow embedding (BFE) structure to infer the voting values ​​of the group points of the source point SP and the target point TP, and adds the group points by using the voting values ​​as weights to extract the initial moving position. In this case, cascade (Concatenate), multi-layer perceptron (MLP) and MAX function can be applied.

[0087] The bidirectional feature detector 130 may detect the initial scene flow through the difference between the extracted initial moving position and the current position.

[0088] The bidirectional feature detector 130 detects an initial scene flow, which is then input into the flow predictor FP, and the initial scene flow is corrected by the features of the bidirectional scene flow to finally detect an accurate scene flow.

[0089] A voting block may be applied to each stage of source points SP and target points TP.

[0090] Hereinafter, the voting operation principle will be described in detail.

[0091] The bidirectional feature detector 130 can collect adjacent point groups in the target frame based on the source frame. All group points have coordinates and features relative to the center of the group.

[0092] The bidirectional feature detector 130 may perform feature transformation of group points through an MLP layer.

[0093] The bidirectional feature detector 130 may calculate a SoftMax value for each group point by using the transformed features of the group points.

[0094] The bidirectional feature detector 130 may assign the calculated SoftMax value as the weight of the group point, and generate the relative coordinates of all the group points as a new vector through a weighted summation scheme. The generated vector may be a scene flow vector.

[0095] Meanwhile, SoftMax is usually used for normalization to express as a uniform probability value. The bidirectional stream embedding structure according to the present disclosure uses SoftMax as a weight, rather than being expressed as a probability value distribution. That is, the weight is calculated based on the features of each group point.

[0096] For the same purpose, Multi-Layer Perceptron (MLP), Linear Transformation, etc. can be used instead of SoftMax, but SoftMax is used as normalization, so when SoftMax is used, outliers (unexpectedly large numbers) can be avoided.

[0097] Fig. 6A and Figure 6B is a diagram showing the source code of the voting function for initial scene flow inference.

[0098] Fig. 6A The model source code is shown, which shows the output part of the stream embedding structure.

[0099] Figure 6B The pattern source code with the voting function applied is shown.

[0100] Figure 7 The invention is a flowchart of an iterative execution process of a bidirectional flow embedding layer in a scene flow estimation method according to an exemplary embodiment of the present disclosure.

[0101] Reference Figure 7 According to an exemplary embodiment of the present disclosure, a scene flow estimation method applies a bidirectional flow embedding layer of an iterative execution structure to a deep neural network model including a dual-point flow network to improve flow prediction accuracy.

[0102] The scene flow estimation method according to an exemplary embodiment of the present disclosure may include a first detection step S710 , a correction step S720 , a second detection step S730 , and an inference step S740 .

[0103] In the first detection step S710, the hierarchical feature detector 110 may receive point cloud data of at least two consecutive image frames. Here, the image frames may include a source image frame and a target image frame. The source image frame may be composed of a set of source points, and the target image frame may be composed of a set of target points.

[0104] The hierarchical feature detector 110 may be configured as PointConv and may detect hierarchical features of point cloud data. The hierarchical features may include staged features from a low stage to a high stage. The low stage may have a low level and a high resolution. The high stage may have a high level and a low resolution.

[0105] In the correction step S720 , the corrector 120 may upsample the features by applying an interpolation scheme to the hierarchical features of the detected point cloud data starting from the high stage, and propagate the upsampled high stage features and scene flow to the low stage.

[0106] In the second detection step S730, the bidirectional feature detector 130 may detect bidirectional scene flow features at each stage of the upsampled source point (SP) and target point (TP). In this case, a flow embedding may be generated. The bidirectional feature detector 130 may repeatedly perform the bidirectional scene flow feature detection process by inputting the detection value.

[0107] In the inference step S740, the inference unit 140 may infer the scene flow of the next image frame by using the bidirectional scene flow feature of each stage of the point cloud data. Meanwhile, in the correction step S720, the corrector 120 may perform a warp on the inference value of the inference unit 140 and use the warped inference value as an input of the hierarchical feature of the point cloud data. Thereafter, these steps may be repeatedly performed.

[0108] Figure 8 The invention is a flowchart including an initial scene flow inference process in a scene flow estimation method according to an exemplary embodiment of the present disclosure.

[0109] Reference Figure 8 , a scene flow estimation method according to an exemplary embodiment of the present disclosure applies an initial flow inference scheme using a voting function to reduce the number of calculation parameters and enhance flow prediction accuracy.

[0110] Reference Figure 8 The scene flow estimation method according to an exemplary embodiment of the present disclosure may include a first detection step S810, a correction step S820, a second detection step S830, an initial scene detection step S840 and an inference step S850.

[0111] In the first detection step S810, the hierarchical feature detector 110 may receive point cloud data of at least two consecutive image frames. Here, the image frames may include a source image frame and a target image frame. The source image frame may be composed of a set of source points, and the target image frame may be composed of a set of target points.

[0112] The hierarchical feature detector 110 may be configured as PointConv and may detect hierarchical features of point cloud data. The hierarchical features may include staged features from a low stage to a high stage. The low stage may have a low level and a high resolution. The high stage may have a high level and a low resolution.

[0113] In the correction step S820 , the corrector 120 may upsample the features by applying an interpolation scheme to the hierarchical features of the detected point cloud data starting from the high stage, and propagate the upsampled high stage features and the scene flow to the low stage.

[0114] In the second detection step S830, the bidirectional feature detector 130 may detect bidirectional scene flow features at each stage of the upsampled source point (SP) and target point (TP). In this case, a flow embedding may be generated. The bidirectional feature detector 130 may repeatedly perform the bidirectional scene flow feature detection process by inputting the detection value.

[0115] In the initial scene detection step S840, the bidirectional feature detector 130 configures a voting block for each stage of the upsampled point cloud data to detect the initial scene flow. Here, the bidirectional feature detector 130 infers the voting values ​​of the grouped group points by applying SoftMax to the flow propagation block of the flow embedding layer, and adds the group points using the corresponding values ​​as weights to extract the initial moving position. Voting can be used for each stage.

[0116] In the inference step S850 , the inferer 140 may infer the scene flow of the next image frame by using the bidirectional scene flow features of each stage of the point cloud data.

[0117] Furthermore, the inferer 140 may infer the scene flow of the next image frame by inputting the bidirectional scene flow features of each stage of the point cloud data into the propagation block to correct the initial scene flow.

[0118] Meanwhile, in the correction step S820, the corrector 120 may perform warping on the inference value of the inference unit 140, and use the warped inference value as an input for extracting hierarchical features of the point cloud data. Thereafter, these steps may be repeatedly performed.

[0119] The above description only illustrates the technical spirit of the present disclosure, and those skilled in the art to which the present disclosure belongs may make various changes, modifications and substitutions without departing from the basic features of the present disclosure. Therefore, the exemplary embodiments and drawings disclosed in the present disclosure are not used to limit but to describe the technical spirit of the present disclosure, and the scope of the technical spirit of the present disclosure is not limited by the exemplary embodiments and drawings.

[0120] As will be appreciated by those skilled in the art, in other exemplary embodiments, for other time periods, etc., steps and / or operations according to the present disclosure may occur in a different order, in parallel, or simultaneously.

[0121] According to an exemplary embodiment, at least some or all of the steps and / or operations may be implemented or performed using commands, programs, and interactive data structures stored in one or more non-transitory computer-readable media and one or more processors driving a client and / or server. As an example, one or more non-transitory computer-readable media may be software, firmware, hardware, and / or any combination thereof. In addition, the functions of the "modules" discussed in this specification may be implemented by software, firmware, hardware, and / or any combination thereof.

[0122] As mentioned above, exemplary embodiments have been described and illustrated in the drawings and the specification. Exemplary embodiments are selected and described in order to explain certain principles of the present disclosure and their practical applications, so that other technical personnel in the art can make and utilize various exemplary embodiments of the present disclosure and various replacements and modifications thereof. As is apparent from the above description, certain aspects of the present disclosure are not limited by the specific details of the examples described herein, and it is therefore expected that other technical personnel in the field will think of other modifications and applications or their equivalents. However, after considering the specification and the drawings, many changes, modifications, variations and other uses and applications of this construction will become apparent to those skilled in the art. All such changes, modifications, variations and other uses and applications that do not depart from the spirit and scope of the present disclosure are considered to be covered by the present disclosure limited only by the appended claims.

Claims

1. A scene flow estimation device, comprising: Controller, including: a hierarchical feature detector configured to detect hierarchical features of point cloud data associated with an image frame; a corrector configured to correct the detected hierarchical features of the point cloud data; a bidirectional feature detector configured to detect a bidirectional scene flow feature of the corrected point cloud data, re-input a detection value, and repeatedly detect the bidirectional scene flow feature; and An inference device is configured to infer a scene flow of a next image frame by using the bidirectional scene flow feature.

2. The scene flow estimation device according to claim 1, wherein: The bidirectional feature detector detects an initial scene flow of the point cloud data by using a predetermined voting block.

3. The scene flow estimation device according to claim 2, wherein: The bidirectional feature detector calculates voting values ​​of group points of the grouping of the point cloud data by applying SoftMax to the voting block, and adds the group points using the voting values ​​as weights to extract an initial moving position for detecting the initial scene flow.

4. The scene flow estimation device according to claim 3, wherein: The inferrer infers the scene flow of the next image frame by correcting the initial scene flow using the bidirectional scene flow feature.

5. The scene flow estimation device according to claim 1, wherein: The corrector applies an interpolation scheme to the detected hierarchical features of the point cloud data and upsamples the hierarchical features.

6. The scene flow estimation device according to claim 1, wherein: The corrector performs warping on the inference value of the inference unit and uses the warped inference value as an input for extracting the hierarchical features of the point cloud data.

7. A scene flow estimation method, comprising: detecting, by a hierarchical feature detector of the controller, hierarchical features of the point cloud data associated with the image frame; Correcting, by a corrector of the controller, the detected hierarchical features of the point cloud data; The bidirectional feature detector of the controller detects the bidirectional scene flow feature of the corrected point cloud data, re-inputs the detection value, and repeatedly detects the bidirectional scene flow feature; as well as The scene flow of the next image frame is inferred by an inference unit of the controller by using the bidirectional scene flow feature.

8. The scene flow estimation method according to claim 7, further comprising: The bidirectional feature detector of the controller detects an initial scene flow of the point cloud data by using a predetermined voting block.

9. The scene flow estimation method according to claim 8, wherein: Detecting the initial flow includes: The bidirectional feature detector calculates voting values ​​of group points of the grouping of the point cloud data by applying SoftMax to the voting block, and adds the group points using the voting values ​​as weights to extract an initial moving position for detecting the initial scene flow.

10. The scene flow estimation method according to claim 9, wherein: Inferring the scene flow of the next image frame includes: The scene flow of the next image frame is inferred by the inference unit of the controller by correcting the initial scene flow using the bidirectional scene flow feature.

11. The scene flow estimation method according to claim 7, wherein: The hierarchical features detected by the calibration include: An interpolation scheme is applied by the corrector of the controller to the detected hierarchical features of the point cloud data and the hierarchical features are upsampled.

12. The scene flow estimation method according to claim 7, wherein: The hierarchical features detected by the calibration include: Warping is performed on the inference value of the inference device by the corrector of the controller, and the warped inference value is used as an input for extracting the hierarchical features of the point cloud data.

13. A non-transitory computer-readable medium containing program instructions executed by a processor, the computer-readable medium comprising: program instructions for detecting hierarchical features of point cloud data associated with an image frame; Program instructions for correcting the detected hierarchical features of the point cloud data; detecting bidirectional scene flow features of the corrected point cloud data, re-inputting detection values, and repeating program instructions for detecting the bidirectional scene flow features; as well as Program instructions for inferring scene flow for a next image frame by using the bidirectional scene flow features.

Images