Method and apparatus for real-time path lookup

Abstract

Methods and apparatus for real-time path lookup are disclosed. A processor-implemented method for real-time path lookup includes combining sensed information from a plurality of sensors at a current point in time with a previous path of a vehicle at a previous point in time and topology information of a topology containing a plurality of nodes at the previous point in time; determining a first distribution region and a second distribution region within the search region by inputting the combined information to a topology-based neural network, where the first distribution region includes the determined feasible path and the second distribution region includes disconnected nodes determined to be reconnected or rewired due to an obstacle; and generating a current path from the target position of the vehicle to a current point in time of the current position of the vehicle using topology information of the current point in time determined based on at least one of the first distribution area and the second distribution area.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2023-0117186 filed on September 4, 2023, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0002] The following description relates to methods and apparatus for real-time path finding. Background Art

[0003] As the demand for autonomous driving technology increases, various technologies for supporting driving have been developed. For example, as a method of driver assistance, a lane departure warning system (LDWS) can determine whether a vehicle deviates from a driving lane, and an adaptive cruise control (ACC) can automatically control the speed of the vehicle while maintaining a predetermined distance from the car in front. In addition, by using detection sensors, image processing sensors, communication devices, etc., an advanced driver assistance system (ADAS) and / or an autonomous driving system (AD) including the above-mentioned technologies can control the operation of the vehicle when the vehicle identifies and determines a driving situation (such as an obstacle or a dangerous situation) while the driver is driving, or can notify the driver of the vehicle. For example, ADAS can provide driving information by finding a path in real time by using a lane of a driving road using image recognition from a camera and / or from pre-built map information. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one general aspect, a processor-implemented method is provided herein, the method comprising: embedding sensing information at a current time point from a plurality of sensors into a previous path at a previous time point of a vehicle and topology information of a topology including a plurality of nodes at the previous time point; determining a first distribution area and a second distribution area within a search area by inputting the embedded sensing information into a topology-based neural network, wherein the first distribution area includes a determined feasible path and the second distribution area includes disconnected nodes determined to be reconnected or rewired due to an obstacle; and generating a current path at the current time point from a target position of the vehicle to a current position of the vehicle using the topology information at the current time point determined based on the determining step.

[0006] The embedding may include: reordering the topology information at the previous time point based on a previous path node corresponding to a previous path at the previous time point; and embedding the sensing information at the current time point into the reordered topology information.

[0007] The reordering of the topology information at the previous time point may include allocating weights to paths included in the topology information at the previous time point based on distances from previous paths at the previous time point.

[0008] The topology-based neural network may include: one or more of a first head, a second head, and a third head, wherein the one or more first heads, second heads, and third heads are each trained to output respective sampling areas corresponding to a first distribution area, a second distribution area, and a third distribution area in response to receiving reordered topology information and sensing information by the topology-based neural network, the third distribution area including determined nodes suitable for pruning to be pruned.

[0009] The method may include training at least one of the first head, the second head, or the third head based on a first loss based on a cross entropy between a first ground truth region corresponding to a training search region and a training sampling region, and a second loss based on a mean square error between a final found previous path and a second ground truth region, the second ground truth region corresponding to a first distribution region for training a topology-based neural network and a second distribution region for training a topology-based neural network.

[0010] The topology information at the previous time point may include information on the extended nodes extended until the previous time point.

[0011] The sensing information at the current time point may include: one or more of: surrounding environment information about surrounding objects of the vehicle and obstacles around the vehicle, the current position of the vehicle, and the target position of the vehicle, and wherein the method may include: receiving the surrounding environment information from a sensor or capturing an image of the surrounding environment information from an image sensor.

[0012] The step of determining the first distribution area and the second distribution area may include: extending one or more of the plurality of nodes according to a determined feasible path found in the first distribution area; and rewiring one or more of the plurality of nodes determined to be disconnected in the second distribution area based on topology information at a previous time point and sensing information at a current time point, wherein the plurality of nodes are determined to be disconnected or newly connected.

[0013] The step of expanding the multiple nodes may include: performing a first sampling non-uniformly on the determined feasible paths in a first distribution area; randomly performing a second sampling on the determined feasible paths in the entire search area; and performing a step of expanding one or more of the multiple nodes based on the first sampling result and the second sampling result.

[0014] In the third distribution area, when it is determined that one or more of the multiple nodes are irrelevant to the vehicle's moving path, the nodes suitable for pruning identified from the topology can be pruned, and the nodes suitable for pruning can be identified from the topology information of the previous time point based on the third distribution area.

[0015] The nodes suitable for pruning may include one or more of: a first node in an area that the vehicle has passed through at a current point in time; and a second node in an area that is irrelevant to a moving path of the vehicle.

[0016] The method may include designating an object observed in the second distribution area as an object of interest by identifying position information of a vehicle changed from a current path at the current time point based on the sensing information at the current time point and the one or more rewired nodes.

[0017] The method may include iteratively finding a current path from a target position of the vehicle to a current point in time of a moving current position of the vehicle as the vehicle moves.

[0018] In a general aspect, a method for training a neural network is provided herein, the method comprising: receiving a training data set, the training data set comprising: sensing information of a vehicle at a previous time point, topological information of a topology including a plurality of nodes at the previous time point, a first area, a second area, and a third area, the first area comprising a feasible path at a current time point corresponding to the topological information at the previous time point, in the second area, a blocked node among the plurality of nodes is determined for rewiring due to an obstacle at the current time point, and in the third area, a node among the plurality of nodes suitable for pruning is to be pruned due to being irrelevant to a moving path of the vehicle at the current time point; and training the neural network to output one or more sampling areas based on the training data set, the sampling areas comprising: a first distribution area containing feasible paths, a second distribution area containing blocked nodes, and a third distribution area containing nodes suitable for pruning.

[0019] The neural network may include: a first head, a second head, and a third head, which are respectively trained to output corresponding sampling areas corresponding to the first distribution area, the second distribution area, and the third distribution area in response to topology information in which nodes are rewired and sensing information at a current time point received by the neural network.

[0020] The training method may include: training at least one of the first head, the second head or the third head based on a first loss and a second loss, the first loss being based on a cross entropy between a first corresponding ground truth region and a training sampling region, the first corresponding ground truth region corresponding to a first distribution region for training a training neural network, a second distribution region for training a training neural network, and a third distribution region for training a training neural network, the training sampling region corresponding to the first distribution region, the second distribution region, and the third distribution region, the second loss being based on a mean square error between a previous path found by the neural network and a second corresponding ground truth region, the second corresponding ground truth region corresponding to the first training distribution region, the second training distribution region, and the third training distribution region.

[0021] In one general aspect, an electronic device is provided herein, the electronic device comprising: a processor configured to execute instructions; a plurality of sensors configured to sense sensing information at a current point in time; and a memory storing instructions, wherein execution of the instructions configures the processor to: embed the sensing information at the current point in time into a previous path of a vehicle at a previous point in time and topology information of a topology including a plurality of nodes at the previous point in time, determine a first distribution area, a second distribution area, and a third distribution area within a search area by inputting the embedded sensing information into a topology-based neural network, wherein the first distribution area includes a determined feasible path, the second distribution area includes disconnected nodes determined to be reconnected or rewired due to obstacles, and the third distribution area includes nodes determined to be suitable for pruning from the topology due to being irrelevant to a moving path of the vehicle; and generate a current path at the current point in time from a target position of the vehicle to a current position of the vehicle using the topology information at the current point in time determined based on the first distribution area, the second distribution area, and the third distribution area.

[0022] The processor may be configured to: expand one or more of the plurality of nodes according to a determined feasible path found in the first distribution area, rewire the one or more plurality of nodes in an area in which one or more of the plurality of nodes are determined to be disconnected or newly connected in the second distribution area based on topological information at a previous time point and sensing information at a current time point, and prune nodes suitable for pruning among the rewired nodes based on topological information from a third distribution area.

[0023] The embedding step may include reordering the topological information of the previous time point based on the previous path node corresponding to the previous path at the previous time point, and embedding the sensing information of the current time point into the reordered topological information, and the topology-based neural network may include a first head, a second head, and a third head, the first head, the second head, and the third head are respectively trained to output corresponding sampling areas corresponding to the first distribution area, the second distribution area, and the third distribution area in response to receiving the reordered topological information and the sensing information of the current time point by the topology-based neural network, and the third distribution area may include determined nodes suitable for pruning.

[0024] At least one of the first head, the second head, or the third head may be trained based on a first loss and a second loss, the first loss being based on a cross entropy between a first ground truth region corresponding to a training search region and a training sampling region, the second loss being based on a mean square error between a final found previous path and a second ground truth region, the second ground truth region corresponding to a first distribution region for training a topology-based neural network and a second distribution region for training a topology-based neural network, and the apparatus may include a vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1A-Figure 1C Example topologies and example methods for real-time path finding are shown in accordance with one or more embodiments.

[0026] Figure 2 An example real-time path finding method is shown in accordance with one or more embodiments.

[0027] Figure 3 Examples of first, second, and third distribution areas determined during a real-time path finding process are shown in accordance with one or more embodiments.

[0028] Figure 4 An example of configuration and operation of an apparatus for real-time path finding according to one or more embodiments is shown.

[0029] Figure 5 Example operations of a topology-based neural network are shown in accordance with one or more embodiments.

[0030] Figure 6 An example method of generating training data in accordance with one or more embodiments is shown.

[0031] Figure 7 Examples of training data are shown in accordance with one or more embodiments.

[0032] Figure 8 An example method of training a neural network in accordance with one or more embodiments is shown.

[0033] Fig. 9 An example device with real-time path finding including an object of interest selection module is shown in accordance with one or more embodiments.

[0034] Fig.10 An example electronic device is shown in accordance with one or more embodiments.

[0035] Throughout the drawings and detailed description, unless otherwise described or provided, the same reference numerals may be understood to refer to the same or similar elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0036] The following detailed description is provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the method, device and / or system described herein will be clear. For example, except for the sequence within the operation and / or the sequence of operations that must occur in a specific order, the sequence within the operation and / or the sequence of operations described herein are merely examples, and are not limited to the sequence set forth herein, but can be clearly changed as after understanding the disclosure of the present application. As another example, except for at least a portion of the sequence within the sequence and / or the sequence within the operation that must occur in a certain order (e.g., a specific order), the sequence of operations and / or the sequence within the operation can be performed in parallel. In addition, in order to increase clarity and brevity, the description of features known after understanding the disclosure of the present application can be omitted.

[0037] The features described herein may be embodied in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have only been provided to illustrate some of the many possible ways to implement the methods, devices, and / or systems described herein that will be apparent after understanding the disclosure of the present application.

[0038] Throughout the specification, when a component or element is described as being "on", "connected to", "bonded to" or "joined to" another component, element or layer, it may be directly (e.g., in contact with another component or element) "on", "connected to", "bonded to" or "joined to" another component, element or layer, or one or more other components, elements, layers may reasonably exist between them. When a component or element is described as being "directly on", "directly connected to", "directly joined to" or "directly joined to" another component or element, there may be no other elements between them. Similarly, expressions such as "between" and "immediately between" and "adjacent to" and "immediately adjacent to" may also be interpreted as described above.

[0039] The terms used herein are only used to describe various examples and should not be used to limit disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. As a non-limiting example, the terms "include", "comprise" and "have" specify the features, quantities, operations, components, elements and / or combinations of the existence of elaboration, but do not exclude the existence or addition of one or more other features, quantities, operations, components, elements and / or combinations of the existence of optional elaboration features, quantities, operations, components, elements and / or combinations of the existence of alternative elaboration. In addition, although an embodiment may be described such that the terms "include", "comprise" and "have" specify the features, quantities, operations, components, elements and / or combinations of the existence of elaboration, there may be other embodiments in which one or more of the features, quantities, operations, components, elements and / or combinations of the existence of elaboration are not present.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which the present disclosure belongs and based on an understanding of the disclosure of the present application. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the disclosure of the present application, and should not be interpreted in an idealized or overly formal sense. The use of the term "may" herein with respect to an example or embodiment (e.g., with respect to what an example or embodiment may include or implement) means that there is at least one example or embodiment that includes or implements such a feature, but all examples are not limited thereto.

[0041] Examples of driving systems may be used to mark lane lines in augmented reality navigation systems for smart vehicles, etc., or to generate visual information for assisting steering of an automated vehicle. In addition, example driving systems may be used to assist safe and comfortable driving by interpreting visual information via a device including an intelligent system, which may include, as a non-limiting example, a display or head-up display (HUD) provided in a vehicle to provide driving assistance or provided in a system that can perform fully automated driving. In a non-limiting example, such a driving system may be or may be applied to an automated vehicle, a smart vehicle, a smart phone, a mobile device, etc.

[0042] Figure 1A-Figure 1C An example topology for real-time path finding and an example method that can be performed according to one or more embodiments are shown. In one non-limiting example, Figure 1A An example of a driving situation of vehicle 115 (e.g., attempting to park in parking space 120 or 125 from road 110) may be shown, while Figure 1B Diagram 130 may illustrate a topology for real-time path finding for vehicle 115 . Figure 1C Diagram 150 may illustrate a collision with an obstacle 155 .

[0043] The vehicle 115 may include all modes and devices of a transportation mechanism for moving a person or object using a drive engine or an engine (such as a car, bus, motorcycle, or truck). In one example, the vehicle 115 may be an intelligent vehicle including an advanced driver assistance system (ADAS) and / or an autonomous driving system (ADS), in which the intelligent vehicle identifies and determines some situations while driving by using sensors, image processors, and / or communication devices, and controls the operation of the intelligent vehicle or notifies the driver of the intelligent vehicle of the situation. ADAS may provide the vehicle 115 with information about the presence of obstacles, surrounding traffic conditions, lane lines of the road 110, and other relevant driving information, which may be identified using images obtained from a camera and / or map information that has been pre-established. In addition, additional driving information including lanes identified in association with the driving road may be provided to the vehicle 115 by a navigation system provided in the vehicle 115. In one example, the vehicle 115 may be referred to as an "autonomous driving vehicle."

[0044] As a non-limiting example, the ADS or ADAS may perform path planning and / or path finding based on information detected by sensors that may include cameras and / or lidar systems to turn the vehicle 115. The search area for the path may be divided into a structured space where lanes and traffic information exist and an unstructured space (such as a parking lot) where lanes or traffic information do not exist, for example.

[0045] Generally, in structured space, paths can be found along lanes or lane boundaries. However, in unstructured space, all drivable spaces typically need to be detected.

[0046] For example, in Figure 1A In the case of a narrow parking space (e.g., parking space 125) where precise control of vehicle 115 may be required, typical path finding may fail if the drivable space is discretized. Therefore, in an unstructured space (such as continuous parking space 120 on the side of road 110), a sampling-based path finding method may be used, which may search all drivable spaces 120 and 125.

[0047] In one example, to efficiently search for a space in which the vehicle 115 can travel, a sampling-based path finding method may be performed by using a data structure such as Figure 1B The topology may include, for example, a tree or graph as shown in FIG. 1 to configure a topology having a sampling area in the path search area. The topology may include, for example, path information corresponding to a previous time point in the form of a tree or graph. The topology may include a plurality of nodes corresponding to each path. For example, Figure 1B As shown in FIG. , when the starting position (q init ) 131 to the target location (q) corresponding to the final destination of the vehicle 115 goal )133 performs sampling to ensure connectivity through extended nodes, a sampling-based path finding method can calculate the best cost path through topology refinement. However, such a typical method of performing sampling on all drivable spaces and configuring topology can ultimately consume a lot of computing time. Since the amount of computation increases exponentially with the increase of the search area, the electronic device in one or more embodiments can use an alternative method of non-uniform sampling in real time starting from an area with a greater probability that an optimal path may exist, rather than performing typical uniform sampling on all available space.

[0048] For example, refer to Figure 1C , when an obstacle 155 appears in a path in the tree of the topology expanded by sampling, since the tree may be difficult to expand, the electronic device may search for a new path to avoid the obstacle 155. For example, the obstacle 155 may include a road structure such as an obstacle and a lane separator installed on a traveling road, and vehicles and pedestrians around the vehicle, but examples are not limited thereto. For example, the new path may include a node q close to the obstacle 155. nearest , node q based on random sampling rand and the node q newly generated due to obstacle 155 new.

[0049] In one example, a method of non-uniformly sampling areas with a greater likelihood of an expected path based on deep learning using surrounding environment information and a training data set of an expected path for a current point in time according to one or more embodiments may be used to find a path in real time.

[0050] Figure 2 An example real-time path finding method is shown in accordance with one or more embodiments.

[0051] Reference Figure 2 In a non-limiting example, the electronic device may search for an expected path at the current time point through operations 210 to 240. Although these illustrated operations may be performed sequentially, the examples do not necessarily have to be performed sequentially. For example, the order of the operations may be changed, and at least two of the operations may be performed in parallel. In an example, as a non-limiting example, the electronic device may be a vehicle (e.g., Figure 1A vehicle 115) or ADS or ADAS.

[0052] In operation 210, the search device may receive sensing information at a current time point from a plurality of sensors. For example, the sensing information at the current time point may include at least one of surrounding environment information including surrounding objects of the vehicle and obstacles around the vehicle, the current position of the vehicle, and the target position of the vehicle. However, these examples are not limited thereto.

[0053] In operation 220, the electronic device may embed the sensing information of the current time point received in operation 210 into the expected path of the vehicle at the previous time point and the topology information of the previous time point. The electronic device may combine (e.g., embed) multiple pieces of information by reordering the topology based on the nodes corresponding to the expected path. The electronic device may reorder the topology information of the previous time point based on the nodes corresponding to the expected path of the previous time point (i.e., the expected path nodes). The electronic device may reorder the topology information of the previous time point by assigning weights to the paths included in the topology information of the previous time point based on the distance from the expected path of the previous time point. In this case, the topology information of the previous time point may include information about nodes that can be extended to the previous time point. The electronic device may embed the sensing information of the current time point into the reordered topology information.

[0054] In operation 230, the embedded information in operation 220 is input into a topology-based neural network as a non-limiting example (e.g., as shown below Figure 4), the electronic device may determine a first distribution area in which a feasible path can be found for the search area. That is, the first distribution area may include an area in which a search for a feasible path is performed. The electronic device may also determine a second distribution area within the search area and / or within the first distribution area, and the second distribution area may include one or more nodes in the topology, which may involve reconnection and / or rewiring of these nodes due to the presence of one or more obstacles. The search area and / or the first distribution area may also include a third distribution area, and since the third distribution area may be irrelevant to the movement path of the vehicle, in the third distribution area, nodes are allowed to be pruned from the topology (i.e., prune-eligible nodes). In this case, a "feasible path" may be understood as an available path for driving a vehicle. That is, in other words, a feasible path may be a path that can be guided or controlled to reach a target location by avoiding one or more obstacles during actual driving of the vehicle without interference (i.e., collision or contact with obstacles or other objects).

[0055] The second distribution area may include information about nodes involved in reconnection and / or rewiring (e.g., rewiring nodes or blocking nodes) in the corresponding area. The third distribution area may include information about unused nodes when setting paths from the topology (in other words, removable nodes (e.g., pruned nodes)). Figure 3 Examples of the first distribution area, the second distribution area, and the third distribution area are described and shown in more detail.

[0056] In an example where one or more distribution areas include a third distribution area, the electronic device may prune nodes that are subject to pruning from topological information at a previous point in time, wherein the nodes are identified among the rewired nodes based on the third distribution area. In this case, the nodes that are subject to pruning may correspond to removable nodes when searching for a path because the nodes are located in an area that is away from the vehicle's driving path and does not correspond to the vehicle's driving path (an area that is unrelated to the vehicle's moving path). The nodes that are subject to pruning may include, for example, nodes that are located in an area that the vehicle has passed through at the current point in time and nodes that are located in an area that is unrelated to the vehicle's moving path. However, the example is not limited thereto.

[0057] In one example, a topology-based neural network may include, for example, at least one of a first head, a second head, and a third head, which are trained to output sampling areas corresponding to one or more distribution areas including the first distribution area, the second distribution area, and the third distribution area, respectively, when the sensing information and the reordered topology information at the current time point are input. In one example, one or more of the first head, the second head, and the third head may be trained based on a first loss and a second loss, the first loss being based on the cross-entropy between the ground truth area corresponding to the corresponding distribution area and the sampling area, and the second loss being based on the mean square error between the ground truth area corresponding to the distribution area and the expected path found.

[0058] A topology-based neural network can be trained to output sampling regions corresponding to the corresponding distribution regions. Figure 5 Example non-limiting operations of a topology-based neural network are described and illustrated in further detail.

[0059] In a non-limiting example, operation 230 may also include an electronic device (e.g., electronic device 400) expanding nodes based on feasible paths found in the first distribution area. That is, the electronic device may expand nodes according to a determined or found feasible path. In one example, the found feasible path may be found in the first distribution area. The electronic device may perform a first sampling unevenly on the feasible paths in the first distribution area. The electronic device may randomly perform a second sampling on the feasible paths in the entire search area. The electronic device may expand the nodes in the topology based on the first sampling result and the second sampling result. Based on the topological information at the previous time point and the sensing information at the current time point, the electronic device may rewire the nodes in the second distribution area where the disconnection or connection of the nodes can be determined as the desired area. In this case, the rewired nodes may correspond to the rewired nodes described above.

[0060] In operation 240, in one example, the electronic device may find an expected path from the target location of the vehicle to the current location of the vehicle using topological information of the current time point determined based on one or more distribution areas as previously discussed above in operation 230. As the vehicle moves, the electronic device may iteratively find an expected path from the target location of the vehicle to the current time point of the moving current location of the vehicle.

[0061] In one example, the electronic device may designate an object observed in the second distribution area as an object of interest by identifying the position information of a vehicle that has changed in the expected path at the current time point based on the sensing information at the current time point and the rewired nodes. The electronic device may collect sensing information based on the object of interest, and may use the sensing information to find the expected path at the current time point. Fig. 9 The method of setting the object observed in the second distribution area as the object of interest is described in more detail.

[0062] Figure 3 An example of a first distribution area, a second distribution area, and a third distribution area determined during a real-time path finding process according to one or more embodiments is shown. Figure 3 In a non-limiting example, two scenarios are shown including a first scenario 310 and a second scenario 330, the first scenario 310 showing a first distribution area including a feasible path 315 according to one embodiment, the second scenario 330 showing a second distribution area 335 where reconnection of nodes in the topology and / or rewiring of nodes is required due to obstacles 333 and a third distribution area 331 where pruning of nodes is allowed.

[0063] In one example, after calculating the expected path from the input information by the sensor, the electronic device can update the path information of the vehicle in driving in real time through dynamic obstacles or newly detected area information. In this case, when calculating the feasible path 315 using the sensing information at the current time point input by the sensor, it may not be possible to select and sample only the distribution of the area required or expected for the update. In this case, due to the newly detected obstacle 333, the area to be updated and the irrelevant area in the search area can be sampled, and unnecessary node expansion may occur in the topology. Because the amount of system memory used due to unnecessary expansion of nodes gradually increases over time, and the number of nodes to be calculated to rewire the nodes also increases, the processing time of the processor for finding the path again may also increase. In addition, this increase in the number of nodes leads to an increase in the allocated memory, and therefore, significant safety issues may occur in systems with high memory constraints.

[0064] In one non-limiting example, the electronic device may perform non-uniform sampling on an area where an optimal path (e.g., a shortest path) may exist based on deep learning, wherein deep learning may generate a greater probability by inputting information obtained by embedding the sensing information at the current time point into the expected path at the previous time point and the topological information at the previous point. In addition, the electronic device may find the optimal path in real time by reducing the number of nodes used for searching, the amount of memory used, and the processing time of the processor by pruning unnecessary nodes in the topology and rewiring the nodes.

[0065] For example, as shown in the first scenario 310, the electronic device may determine a first distribution area that includes a feasible path 315 in a topology generated during an original path planning process from a starting position 311 of the vehicle 115 to a target position 313 corresponding to a destination. Since the first distribution area includes the feasible path 315, the first distribution area may be referred to as a feasible path distribution area.

[0066] In addition, as shown in the second scenario 330, the electronic device may determine a second distribution area 335 in which nodes may be desirably reconnected or rewired in the topology due to the obstacle 333 (i.e., disconnected nodes, rewired nodes, or blocked nodes). The second distribution area 335 may correspond to an area in which, by considering the topology according to information at a previous time point and changed surrounding information at a current time point (e.g., information about the obstacle 333), nodes in the topology that collide with the obstacle 333 need to be disconnected or new nodes that avoid the obstacle 333 need to be connected. Since the nodes in this area may be rewired, the second distribution area 335 may be referred to as a rewiring node distribution area. The second distribution area 335 may be used as sampling information for selecting nodes whose positions need to be modified in the determined topology at the previous time point according to information at the previous time point and / or reconnecting nodes with modified positions.

[0067] In addition, as shown in the diagram 330, the electronic device may determine a third distribution area 331 in which the node may be removed from the topology because the node is irrelevant to the moving path of the vehicle 115. Since the third distribution area 331 is an expected area in which the vehicle 115 cannot move in the topological structure in the form of a tree (in other words, a pruning target area), the third distribution area 331 may be referred to as a pruning node distribution area, for example.

[0068] Figure 4 An example device for real-time path finding according to one or more embodiments is shown. Figure 4 , in a non-limiting example, shows the structure of the electronic device 400. Figure 4 A portion shown as a dotted line in FIG. 1 may represent a configuration that may be optionally executed.

[0069] In one example, the electronic device 400 may receive the current time point T from multiple sensors. nThe sensing information 401 is the information observed by various sensors installed in the vehicle during the driving of the vehicle, so the sensing information 401 may be referred to as observation information. The sensing information may include, for example, the current time point T n The present invention also includes surrounding environment information of the vehicle (such as surrounding objects of the vehicle and obstacles around the vehicle), the current position of the vehicle, and the target position of the vehicle, but is not limited thereto.

[0070] The electronic device 400 can be based on the previous time point T n-1 The node pair corresponding to the expected path (or optimal path) 403 at the previous time point T n-1 The topology information 405 is reordered 410 and two pieces of information (eg, the previous time point T n-1 The expected path 403 and the previous time point T n-1 The topology information 405 of the previous time point is embedded to generate the reordered topology information 415. The reordered topology information 415 can be, for example, a map including an expansion tree through sampling, but is not limited thereto. In this case, the previous time point T n-1 Can be compared with the current time point T n The previous time point corresponds.

[0071] The electronic device 400 can be based on the previous time point T n-1 The distance of the expected path 403 is assigned a weight to the path included in the previous time point T n-1 The topology information 405 of the previous time point is reordered. n-1 When the distance of the expected path 403 is closer to the predetermined standard, the electronic device may assign a large weight (eg, "1") to the nodes included in the previous time point T n-1 The path in the topology information 405 (such as the reordered topology information 415) is compared with the previous time point T n-1 When the distance of the expected path 403 is further away from the predetermined standard, the electronic device may assign a small weight (eg, "4") to the path included in the previous time point T n-1 The path in the topology information 405 (such as the reordered topology information 415). n-1 The topological information may include information about the extension to the previous time point T n-1 In addition, the topology information may include information about the vehicle state (eg, posture, speed, and acceleration) as node information.

[0072] The electronic device 400 can use the current time point T nThe embedded information obtained by embedding the sensing information 401 into the reordered topology information 415 is input to the topology-based neural network 430. In this case, the topology-based neural network 430 may be a neural network trained to classify the search area into a plurality of distribution areas for sampling based on the input information to find a path. The plurality of distribution areas for sampling may include, for example, a first distribution area 491, a second distribution area 493, and a third distribution area 495 in the map 490.

[0073] In a non-limiting example, the topology-based neural network 430 may include a first head 431, a second head 433, and a third head 435, each of which is trained to output a sampling area corresponding to a first distribution area 491, a second distribution area 493, and a third distribution area 495 as the reordered topology information 415, and the current time point T n The sensing information 401 is input into the topology-based neural network 430. The first head 431 may correspond to a neural network trained to output a sampling area that may include a feasible path corresponding to the first distribution area 491. The second head 433 may correspond to a neural network trained to output a sampling area that may correspond to the second distribution area 493. The third head 435 may correspond to a neural network trained to output a sampling area that may correspond to the third distribution area 495. The first head 431 may be used as a non-uniform sampler for quickly finding an expected path from a global perspective. The second head 433 may provide information about an area where nodes need to be disconnected or connected in the topology by considering the changed surrounding environment information when the vehicle is traveling. The node rewiring process of the second head 433 may be performed in parallel with the node expansion 450 process, and may be effectively applied to quickly find another expected path. The third head 435 may provide information about a sampling area corresponding to the third distribution area 495, which is at the current time point T n No updates of nodes or nodes passing through the zone are required.

[0074] In one example, the first head 431, the second head 433, and the third head 435 may be trained based on a first loss based on a cross entropy between a ground truth region corresponding to distribution regions corresponding to the first head 431, the second head 433, and the third head 435, respectively, and a second loss based on a mean square error between a ground truth region corresponding to distribution regions corresponding to the first head 431, the second head 433, and the third head 435, respectively, and a searched expected path.

[0075] In one example, the electronic device 400 may expand nodes along the feasible paths found in the first distribution area 491. The electronic device 400 may perform a first sampling 440 on the feasible paths non-uniformly in the first distribution area 491. The electronic device 400 may randomly (or uniformly) perform a second sampling 445 on the feasible paths in the entire search area. The electronic device 400 may perform a node expansion 450 process based on the results of the first sampling 440 and the second sampling 445 to expand the nodes of the topology.

[0076] The electronic device 400 may be based on the previous time point T n-1 The topology information 405 and the current time point T n The node rewiring 460 process is performed based on the sensing information 401 to rewire nodes (eg, rewiring nodes) in an area of ​​the second distribution area 493 where nodes need to be disconnected or newly connected.

[0077] In addition, the electronic device 400 may perform a node pruning 470 process to prune nodes rewired by the node rewiring 460 process according to the previous time point T n-1 The topology information 405 is pruned for nodes that are subject to pruning, wherein the nodes that are subject to pruning are identified based on the third distribution area 495 .

[0078] The electronic device 400 may reduce the amount of memory usage by pruning inefficient nodes in the node pruning topology by pruning nodes corresponding to the third distribution area 495 and reconnecting the nodes.

[0079] The electronic device may use the current time point T generated by the node expansion 450 process, the node rewiring 460 process, and the node pruning 470 process. n The topological information 480 is used to generate (ie, find) the current time point T with the best cost from the target position of the vehicle to the current position of the vehicle. n The expected path (or optimal path) 485.

[0080] Figure 5 Example operations of a topology-based neural network according to one or more embodiments are shown. Figure 4 , in a non-limiting example, a diagram 500 is shown for describing the structure and operation of a topology-based neural network 430 according to one embodiment.

[0081] In one example, when an input grid map 510 is input, the topology-based neural network 430 may output an output grid map 530 from the input grid map 510 .

[0082] In this case, the input grid map 510 may include, for example,n-1 The corresponding reordered topology information 415, 2-channel grid map 511, represents the root q k The 1-channel grid map 513 and the pose q representing the target position g The 1-channel grid map 515 is compared with the previous time point T n-1 The corresponding re-ordered topology information 415 includes an expanded tree corresponding to the re-ordered topology information 415 .

[0083] The topology-based neural network 430 may include, for example, a convolutional block consisting of a convolutional layer, a pooling layer, and a non-pooling layer.

[0084] For example, each of the three heads of the topology-based neural network 430 may be trained by optimization of the loss function in Equation 1 as shown below.

[0085] Equation 1:

[0086] Loss = L wce (M label ,M out )+L wmse (M label ,M out )

[0087] In this case, M label M may represent the ground truth region corresponding to the ground truth label, and M out It can correspond to the sampling area output by each head. In addition, the first term L wce (M label ,M out ) may represent the weighted grid-by-cell cross entropy between the ground truth region and the sampled region corresponding to the corresponding distribution region corresponding to the corresponding head. The second term L wmse (M label ,M out ) may represent the grid-by-cell mean square error between the ground truth region corresponding to the corresponding distribution region corresponding to the corresponding head and the expected path output by each head.

[0088] Since trajectory information is not included in most grids of the label grid map, for example, when a grid cell includes trajectory information, the weight W ce and W mse Can be multiplied by the loss function.

[0089] The above equation 1 can also be expressed by equation 2 as shown below.

[0090] Equation 2:

[0091]

[0092] Each of the three heads of the topology-based neural network 430 may calculate a cross entropy loss value L for each cell g in the grid G ce (g), and can be obtained by converting the weight value W ce Multiply by the cross entropy loss value L ce The value obtained by (g) is consistent with that obtained by taking L as the grid-by-grid mean square error mse (g) Multiply by the weight value W mse The obtained values ​​are added together to calculate the final loss.

[0093] In this case, including L ce The cross entropy term in (g) can be designed to classify the location information of the track on the output grid map. mse The mean square error loss term of (g) can be designed to regress the direction information of the track on the grid map. The electronic device can ce and W mse ) is weighted to the final loss as described above to infer the sampling information about the position information and orientation information.

[0094] The output grid map 530 may include, for example, n The corresponding expected path 531 and the cosine of the expected path ( cos ) posture information 533 and the sinusoidal posture information 535 of the expected path.

[0095] The output grid map 530 output by the topology-based neural network 430 can be input to an arbitrary time planning algorithm 550. In one example, the “predicted credible trajectory ” is the prediction result of the credible trajectory distributed by the topology-based samples. In one example, “credible trajectory π c，k+1 (q eg0 ,q k+1 )” corresponds to the optimal path searched at the current time point, where “current tracking” can indicate that it is at the current time point T n is selected and tracked, where q ego indicates the location of the autonomous vehicle, and q k+1 Indicator root q kPositions moved over time. In a non-limiting example, the arbitrary time planning algorithm 550 may provide trajectory control information based on the output grid map 530 to the controller 570. The arbitrary time planning algorithm 550 may be an algorithm for generating control values ​​before the path search is completed by following the point with the most useful cost before reaching the target point among the nodes undergoing the tree search before finding the best path. In one example, the "predicted credible trajectory" reflected in the arbitrary time planning algorithm 550 may cause the vehicle (system) to travel along a credible trajectory.

[0096] For example, in a parking situation, it may take several seconds to fully perform the best path finding, and the arbitrary time planning algorithm 550 can be used to reduce the time the vehicle is stopped and waiting. The arbitrary time planning algorithm 550 can manage the tree to be in an optimal state while the vehicle is driving.

[0097] In one example, the controller 570 can be logic configured to control a vehicle or system and can generate trajectory information generated by a path planning algorithm, wherein the trajectory information includes one or more of (vehicle's) acceleration or deceleration values, steering angles, actuator torques of the robot, or a position for the electronic device to follow.

[0098] More specifically, the arbitrary time planning algorithm 550 may correspond to one of the sampling-based path planning methods for generating an optimal path by following at least a portion of the calculated tree until the current time point before generating a feasible path. In one example, the electronic device may use a random sampling-based path planner algorithm instead of the arbitrary time planning algorithm 550.

[0099] Furthermore, the output grid map 530 may be a portion of the input grid map 510 and iteratively input to the topology-based neural network 430 .

[0100] Figure 6 An example method of generating training data according to one or more embodiments is shown, and Figure 7 Example training data processing is shown in accordance with one or more embodiments.

[0101] Reference Figure 6 , in a non-limiting example, a generating device 600 for generating training data for a topology-based neural network 430 is shown. Figure 7 , in one non-limiting example, input data 710 , label data 730 , an input image 750 , and an expected path 770 corresponding to the input image 750 are shown.

[0102] In one example, the generating device 600 may obtain a first region (e.g., a feasible path region) 640, a second region (e.g., a rewiring region) 620, and a third region (e.g., a pruning region) 630 as ground truth values ​​and may manage the ground truth values ​​as a data set, the first region 640 corresponding to a first distribution region that can be used to train the topology-based neural network 430 at the current time point, the second region 620 corresponding to a second distribution region at the current time point, and the third region 630 corresponding to a third distribution region at the current time point. In one example, the generating device 600 may be one of a training device, an electronic device, or another device.

[0103] Reference Figure 7 , in one example, Figure 6 The generating device 600 may receive a data including a previous time point T n-1 The sensing information 601 and the previous time point T n-1 The input data 710 of the topology information 603 .

[0104] Reference Figure 6 In one example, the generating device 600 may determine the path planner 610 based on sampling including the current time point T n The current time point T of the topology information 613 n The expected path information (or optimal path information) 615 of the current time point T n The topology information 613 can be based on the previous time point T n-1 The sensing information 601 and the previous time point T n-1 Topology information 603.

[0105] The generating device 600 may be based on the current time point T n The expected path information determined by the method generates training data including a first area, a second area, and a third area, wherein the first area includes the area corresponding to the current time point T n The feasible path corresponding to the expected path is rewired in the second region, and the nodes corresponding to the topological information at the current time point Tn are rewired in the third region. n Be pruned.

[0106] More specifically, refer to Figure 7 , since the data generation process does not require real-time response, the generation device may execute the path planner algorithm (e.g., arbitrary time planning algorithm 550) based on random sampling during a period sufficient to fully cover the entire search area and calculate the best expected path. In this process, the generation device 600 may use the current time point T n The state information (such as label data 730) of the nodes corresponding to the expected path determined by the topology information is labeled and used as the current time point Tn The first region (feasible path region) 640 of . For example, the generating device may tag and use the state information of the nodes that are rewired and / or reconnected in the tree topology as the current time point T n In addition, the generating device 600 may label the area where the node is allowed to be pruned in the tree topology as the current time point T n The third region (pruning region) 630. For all nodes after obtaining the expected path 770, the generating device may perform reverse scoring and labeling on nodes that contribute to reducing the cost for calculating the optimal expected path and nodes that succeed in the above-mentioned node expansion.

[0107] The training device may use the input data 710 and the label data 730 to train the topology-based neural network to output an expected path 770 corresponding to the input image 750 .

[0108] Figure 8 An example training method for a neural network according to one or more embodiments is shown. Figure 8 In a non-limiting example, a diagram 800 is shown, which illustrates the use of the above reference Figure 6 The training data set 810 generated by the training device trains the topology-based neural network 430 process.

[0109] In one example, a generating device is used to generate Figure 6 The data of the previous time point T n-1 The sensing information 601 includes the previous time point T n-1 The input data 710 of the topology information 603, the current time point T generated based on the input data 710 n The first area (feasible path area) 640, the current time point T n The second region (rewiring region) 620 and the current time point T n The third region (pruned region) 630 is used as a training data set 810 for training the topology-based neural network 430 .

[0110] In this case, the sensing information at the current time point and the current time point T n The topological information of can be used as input data 710 instead of the previous time point T n-1 The sensing information 601 and the previous time point T n-1 In this case, the next time point T generated based on the input data 710 n+1 The first area (feasible path area) of n+1 The second region (rewiring region) and the next time point T n+1The third region (pruned region) may be used as a training data set 810 for training the topology-based neural network 430 .

[0111] In one example, the training device may train the topology-based neural network 430 to infer the first distribution area 491 , the second distribution area 493 , and the third distribution area 495 based on the training data set 810 collected during the above-described data generation process.

[0112] The topology-based neural network 430 may include, for example, one or more of a first head, a second head, and a third head, which are trained to: n The sensing information 601 and the node at the previous time point T n-1 When the rewired topology information 603 is input to the topology-based neural network 430, sampling regions corresponding to the first distribution region 491, the second distribution region 493, and the third distribution region 495 are output. At least one of the first head, the second head, and the third head may be trained based on a first loss based on a cross entropy between corresponding sampling regions corresponding to the first distribution region 491, the second distribution region 493, and the third distribution region 495 and corresponding ground truth regions (e.g., the first region 640, the second region 620, and the third region 630) corresponding to the first distribution region 491, the second distribution region 493, and the third distribution region 495, and a second loss based on a mean square error between an expected path found by the topology-based neural network 430 and corresponding ground truth regions corresponding to the first distribution region 491, the second distribution region 493, and the third distribution region 495.

[0113] The training device may receive a training data set 810, wherein the training data set 810 includes a previous time point T n-1 The sensing information 601 of the previous time point T n-1 The topology information 603, the first area 640 (eg, the feasible path area T n ), the second region 620 (eg, the rewiring region T n ) and the third region 630 (eg, the trimmed region T n ), the first region 640 includes the region corresponding to the previous time point T n-1 The topology information corresponding to the current time point T n In the second region 620, the node is n The obstacles need to be reconnected or rewired. In the third area 630, since the node is n The movement paths of the vehicles are irrelevant, so nodes are allowed to be pruned.

[0114] Based on the training data set 810, the topology-based neural network 430 can be trained by the training device to output at least one sampling area of ​​the first distribution area 491, the second distribution area 493 and the third distribution area 495, wherein the first distribution area 491 includes feasible paths, the second distribution area 493 includes reconnected or rewired nodes, and the third distribution area 495 includes pruned nodes.

[0115] Fig. 9 An example device for real-time path finding including an object of interest selection module according to one or more embodiments is shown. Fig. 9 In one non-limiting example, diagram 900 illustrates the structure of an electronic device 400 (eg, a multi-objective sample distributor path planner) that may include an object of interest selection module 910 .

[0116] In one example, the object of interest selection module 910 may select an object of interest that may be desirable for adaptive cruise control (ACC) or automatic emergency braking (AEB) as the object of interest by considering a driving path of the vehicle.

[0117] In one example, as described above, the object of interest selection module 910 may select the object of interest based on the current time point T n The sensing information 401 and the rewiring node identification in the second distribution area are from the current time point T n The object of interest selection module 910 can select the object of interest by assigning and managing the object observed in the second distribution area as the object of interest, which is obtained by identifying the previous time point T to be changed. n-1 The location information of the expected path (wherein the expected path has been generated) is further considered to be relevant to the object of interest.

[0118] Fig.10 An example electronic device according to one or more embodiments is shown. Fig.10 In a non-limiting example, an electronic device 1000 according to one embodiment may include a plurality of sensors 1010, a processor 1030, a memory 1050, and an output device 1070. In one example, the electronic device 1000 may be a vehicle or an ADS or ADAS of such a vehicle, but the example is not limited thereto.

[0119] The plurality of sensors 1010 may sense sensing information at the current time point. The plurality of sensors 1010 may include, for example, a monochrome camera, a visual sensor, an image sensor, and / or a capture device including a device performing a similar function to them, as well as a gyroscope sensor, a lidar sensor, a radar, and / or an odometer, but the example is not limited thereto.

[0120] Processor 1030 (or multiple processors) can execute instructions (e.g., code and / or programs), and / or can control other operations or functions of electronic device 1000 and operations of a vehicle (e.g., vehicle 115), and may include, for example, any one or a combination of two or more of a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and a tensor processing unit (TPU), but is not limited to the above examples.

[0121] The processor 1030 may embed the expected path of the vehicle at the previous time point and the topological information at the previous time point into the sensing information at the current time point sensed by the plurality of sensors 1010. By inputting the embedded information into the topology-based neural network, the processor 1030 may determine the search area to find the path as at least one of a first distribution area, a second distribution area, and a third distribution area, the first distribution area including feasible paths, the second distribution area in which nodes need to be reconnected or rewired in the topology due to obstacles, and in the third distribution path, since the nodes are irrelevant to the moving path of the vehicle, the nodes are allowed to be pruned from the topology. In this case, the topology-based neural network may include, for example, at least one of a first head, a second head, and a third head, which are trained to output sampling areas corresponding to the first distribution area, the second distribution area, and the third distribution area including removable nodes, respectively, when the reordered topological information and the sensing information at the current time point are input. At least one of the first head, the second head, and the third head may be trained based on a first loss and a second loss, the first loss being based on a cross entropy between a ground truth region corresponding to at least one distribution region and a sample region, and the second loss being based on a mean square error between a ground truth region corresponding to at least one distribution region and the expected path to be found. The neural network may be, for example, a convolutional neural network (CNN), but is not limited thereto. The CNN may be trained to identify both a bounding box of a lane line (or road sign) to be detected in an input image and a type of lane line (or road sign) to be detected in an input image. The processor 1030 may use the determined topological information at the current time point based on at least one distribution region to find an expected path from a target position of the vehicle to a current position of the vehicle.

[0122] The memory 1050 may include computer-readable instructions. The processor 1030 may be configured to execute computer-readable instructions (such as computer-readable instructions stored in the memory 1050), and by executing the computer-readable instructions, the processor 1030 is configured to perform one or more or any combination of the operations and / or methods described herein. The memory 1050 may be a volatile memory or a non-volatile memory.

[0123] The memory 1050 may store parameters of a topology-based neural network pre-trained to determine at least one distribution area. The processor 1030 may use the topology-based neural network to which the parameters stored in the memory 1050 are applied to find the expected path at the current time point from the input data. In addition, the memory 1050 may store various information generated during the processing of the above-mentioned processor 1030. In addition, the memory 1050 may store various data and programs. The memory 1050 may include a volatile memory or a non-volatile memory. The memory 1050 may include a large-capacity storage medium (such as a hard disk) for storing various data.

[0124] In one example, the output device 1070 may output the expected path at the current time point found by the processor 1030. For example, the output device 1070 may be an output interface or a display device. When the output device 1070 is a display or a head-up display (HUD), the output device 1070 may display or project the expected path at the current time point found by the processor 1030 on the input image or the navigation image, respectively. In addition, when it is predicted that a driving lane or lane needs to be changed based on the expected path at the current time point found by the processor 1030, the output device 1070 may display the direction to which the driving lane is changed.

[0125] The disclosed processors, memories, vehicles, vehicles 115, electronic devices, controllers 570, generating devices 600, multiple sensors 1010, processors 1030, memories 1050, and output devices 1070 described herein and described herein with reference to FIGS. 1-10 are implemented or represent hardware components by hardware components. As described above, or in addition to the above description, examples of hardware components that can be used to perform the operations described in this application include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application, where appropriate. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond in a defined manner and execute instructions to achieve a desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer can execute instructions or software (such as, operating system (OS) and one or more software applications running on OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component or two or more hardware components. As described above, or in addition to the description above, the example hardware components may have any one or more of different processing configurations, where examples of different processing configurations include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0126] Figure 1 to Fig.10 The method for performing the operations described in the present application shown in is performed by computing hardware (e.g., by one or more processors or computers), and the computing hardware is implemented as described above to implement instructions or software for performing the operations performed by the method described in the present application. For example, a single operation or two or more operations can be performed by a single processor or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can perform a single operation or two or more operations.

[0127] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform methods as described above may be written as computer programs, code segments, instructions, or any combination thereof, to be used to individually or collectively instruct or configure one or more processors or computers as machines or special-purpose computers for performing operations performed by hardware components and methods as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by one or more processors or computers. In another example, the instructions or software include high-level code executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions herein, and the block diagrams and flow charts shown in the accompanying drawings disclose algorithms for performing operations performed by hardware components and methods as described above.

[0128] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media, rather than the signal itself. As described above, or in addition to the above description, examples of non-transitory computer-readable storage media include any one or more of the following: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH, BD-RE, Blu-ray or optical disk storage device, hard disk drive (HDD), solid state drive (SSD), flash memory , card memory (such as, multimedia mini card or card (for example, secure digital (SD) or extreme digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and / or any other device configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner, and provide the instructions or software and any associated data, data files and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed by one or more processors or computers in a distributed manner.

[0129] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered only in a descriptive sense and not for limiting purposes. The description of the features or aspects in each example should be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described system, architecture, device, or circuit are combined in a different manner, and / or the components in the described system, architecture, device, or circuit are replaced or supplemented by other components or their equivalents, suitable results may be achieved.

[0130] Therefore, the scope of the disclosure also includes the claims and their equivalents in addition to the disclosure above and all the accompanying drawings, that is, all changes within the scope of the claims and their equivalents should be construed as being included in the disclosure.

Claims

1. A processor-implemented method for real-time path finding, the method comprising: combining sensing information from a plurality of sensors at a current time point with a previous path of the vehicle at a previous time point and topology information of a topology including a plurality of nodes at the previous time point; determining a first distribution area and a second distribution area within the search area by inputting the combined information into a topology-based neural network, wherein the first distribution area includes the determined feasible path and the second distribution area includes disconnected nodes determined to be reconnected or rewired due to obstacles; and A current path at the current time point from the target position of the vehicle to the current position of the vehicle is generated using the topology information at the current time point determined based on at least one of the first distribution area and the second distribution area.

2. The method according to claim 1, wherein: The steps of combination include: reordering the topology information at the previous time point based on the previous path nodes corresponding to the previous path at the previous time point; and The sensing information at the current time point is combined with the reordered topological information.

3. The method according to claim 2, wherein: The step of reordering the topology information at the previous time point includes assigning weights to paths included in the topology information at the previous time point based on distances from previous paths at the previous time point.

4. The method according to claim 2, wherein: The topology-based neural network includes one or more of a first head, a second head, and a third head, which are respectively trained to output corresponding sampling areas corresponding to a first distribution area, a second distribution area, and a third distribution area in response to receiving reordered topology information and sensing information by the topology-based neural network, and the third distribution area includes determined nodes suitable for pruning to be pruned.

5. The method according to claim 4, further comprising: At least one of the first head, the second head, and the third head is trained based on: a first loss, based on the cross entropy between a first ground truth region corresponding to the training search region and the training sampled region; as well as A second loss based on a mean squared error between the final found previous path and a second ground truth region corresponding to the first distribution region for training the topology-based neural network and the second distribution region for training the topology-based neural network.

6. The method according to claim 1, wherein: The topology information at the previous time point includes information on the extended nodes extended until the previous time point.

7. The method according to claim 1, wherein: The sensing information at the current time point includes one or more of: surrounding environment information about surrounding objects of the vehicle and obstacles around the vehicle, the current position of the vehicle, and the target position of the vehicle, and Wherein, the method further comprises: The surrounding environment information is received from the plurality of sensors or an image of the surrounding environment information is captured from an image sensor included in the plurality of sensors.

8. The method according to claim 1, wherein: The step of determining the first distribution area and the second distribution area comprises: expanding one or more of the plurality of nodes according to the determined feasible path found in the first distribution area; and One or more of the plurality of nodes determined to be disconnected are rewired in the second distribution area based on the topology information at a previous time point and the sensing information at a current time point.

9. The method according to claim 8, wherein: The step of expanding the one or more of the plurality of nodes comprises: performing first sampling non-uniformly on the determined feasible paths in the first distribution area; randomly performing a second sampling on the determined feasible paths in the entire search area; and The one or more of the plurality of nodes are expanded based on the first sampling result and the second sampling result.

10. The method according to claim 8, wherein: In a third distribution area, upon determining that the one or more of the plurality of nodes are not related to a movement path of the vehicle, pruning the identified nodes suitable for pruning from the topology, and Therein, nodes suitable for pruning are identified from topological information at a previous time point based on the third distribution area.

11. The method according to claim 10, wherein: Nodes suitable for pruning include one or more of the following: the first node in the area that the vehicle has passed through at the current time point; and A second node in an area that is not related to the movement path of the vehicle.

12. The method according to claim 8, further comprising: An object observed in the second distribution area is designated as an object of interest by identifying position information of a vehicle changed from a current path at the current time point based on the sensing information at the current time point and the one or more rewired nodes.

13. The method according to claim 1, further comprising: As the vehicle moves, the current path from the target position of the vehicle to the current point in time of the vehicle's moving current position is iteratively searched.

14. A method for training a neural network, the method comprising: receiving a training data set, the training data set comprising: sensing information of a vehicle at a previous time point, topology information of a topology including a plurality of nodes at the previous time point, a first area, a second area, and a third area, the first area comprising a feasible path at a current time point corresponding to the topology information at the previous time point, in the second area, blocked nodes among the plurality of nodes are determined for rerouting due to obstacles at the current time point, and in the third area, nodes among the plurality of nodes suitable for pruning are to be pruned due to being irrelevant to a moving path of the vehicle at the current time point; and The neural network is trained based on the training data set to output one or more sampling regions, wherein the one or more sampling regions include: a first distribution region containing feasible paths, a second distribution region containing blocked nodes, and a third distribution region containing nodes suitable for pruning.

15. The training method according to claim 14, wherein: The neural network includes a first head, a second head and a third head, which are respectively trained to output corresponding sampling areas corresponding to the first distribution area, the second distribution area and the third distribution area in response to topology information in which nodes are rewired and sensing information at a current point in time received by the neural network.

16. The training method according to claim 15, further comprising: At least one of the first head, the second head, and the third head is trained based on: a first loss based on a cross entropy between a first corresponding ground truth region and a training sample region, the first corresponding ground truth region corresponding to a first training distribution region for training the neural network, a second training distribution region for training the neural network, and a third training distribution region for training the neural network, the training sample region corresponding to the first distribution region, the second distribution region, and the third distribution region; and A second loss is based on a mean square error between a previous path found by the neural network and a second corresponding ground truth region, the second corresponding ground truth region corresponding to the first training distribution region, the second training distribution region, and the third training distribution region.

17. An electronic device, comprising: a processor configured to execute instructions; A plurality of sensors are configured to sense sensing information at a current time point; as well as A memory storing instructions, wherein execution of the instructions configures the processor to: combining the sensing information at the current time point with a previous path of the vehicle at a previous time point and topology information of a topology including a plurality of nodes at the previous time point, determining a first distribution area, a second distribution area, and a third distribution area within the search area by inputting the combined information into a topology-based neural network, wherein the first distribution area includes the determined feasible path, the second distribution area includes disconnected nodes determined to be reconnected or rewired due to obstacles, and the third distribution area includes nodes determined to be suitable for pruning from the topology due to being irrelevant to the movement path of the vehicle, and A current path at the current time point from the target position of the vehicle to the current position of the vehicle is generated using the topology information at the current time point determined based on the first distribution area, the second distribution area, and the third distribution area.

18. The electronic device according to claim 17, wherein: The processor is also configured to: expanding one or more of the plurality of nodes according to the determined feasible path found in the first distribution area, rewiring one or more nodes among the plurality of nodes determined to be disconnected or newly connected in the second distribution area based on the topology information at a previous time point and the sensing information at a current time point, and Among the rewired nodes, nodes suitable for pruning are pruned from the topology information based on the third distribution area.

19. The electronic device according to claim 17, wherein: The steps of combination include: reordering the topology information at the previous time point based on the previous path nodes corresponding to the previous path at the previous time point; and Combine the sensing information at the current time point with the reordered topological information, Among them, topology-based neural networks include: The first head, the second head and the third head are respectively trained to output corresponding sampling areas corresponding to the first distribution area, the second distribution area and the third distribution area in response to receiving reordered topology information and sensing information at a current time point by a topology-based neural network, and the third distribution area includes determined nodes suitable for pruning.

20. The electronic device according to claim 19, wherein: At least one of the first head, the second head, and the third head is trained based on: a first loss, based on the cross entropy between a first ground truth region corresponding to the training search region and the training sampled region; as well as a second loss based on a mean squared error between the final found prior path and a second ground truth region corresponding to the first distribution region for training the topology-based neural network and the second distribution region for training the topology-based neural network, and Wherein, the electronic device includes a vehicle.

Images