Method, device, storage medium for modeling an object, and vehicle control method

Abstract

The application provides a method, device, storage medium and vehicle control method for modeling an object, by obtaining a first set of point cloud frames associated with a surrounding environment collected by a perception system; generating a hierarchical representation of an object in the surrounding environment according to the first set of point cloud frames, the hierarchical representation including a first representation of the object; obtaining a second set of point cloud frames associated with the surrounding environment collected by the perception system; determining a set of points associated with the object in each frame of point cloud of the second set of point cloud frames and a first subset and a second subset of the set of points; determining a first confidence of a first part associated with the first subset and a second confidence of a second part associated with the second subset according to the first subset and the second subset; and determining whether to change the hierarchical representation of the object according to the first confidence and the second confidence. The application optimizes the representation of the object according to observation dynamics, so that the representation of the object is more accurate and stable with observation.

Description

Technical Field

[0001] This application relates to the processing of sensor data, and more particularly to a method, apparatus, storage medium, and vehicle control method for modeling objects. Background Technology

[0002] With the development of autonomous driving technology, autonomous vehicles have already been applied in freight and passenger transport. When driving, autonomous vehicles typically perceive their surroundings through their own sensors, such as LiDAR systems and cameras. The autonomous driving system within the vehicle generates representations of objects in the environment (e.g., bounding boxes) based on the data perceived by the sensors, expressing the objects' poses and / or extents, and determining their motion states. The autonomous driving system then makes decisions and plans for the vehicle's movement, ultimately controlling the vehicle to drive according to these decisions and plans.

[0003] In order to make reasonable plans, autonomous driving systems need to generate accurate representations of objects in the environment in order to accurately determine the motion state of the objects. Summary of the Invention

[0004] This application provides a method, apparatus, storage medium, and vehicle control method for modeling objects, which dynamically models objects in the environment surrounding a vehicle (especially an autonomous vehicle) to generate an accurate representation of the object so as to accurately determine the motion state of the object.

[0005] In a first aspect, this application provides a method for modeling objects, including:

[0006] Acquire the first set of point cloud frames associated with the surrounding environment collected by the sensing system;

[0007] Generate a hierarchical representation of objects in the surrounding environment based on the first set of point cloud frames. The hierarchical representation includes a first representation of the objects.

[0008] Acquire a second set of point cloud frames associated with the surrounding environment, collected by the sensing system;

[0009] Determine a set of points associated with the object in each frame of the second set of point cloud frames, as well as a first subset and a second subset of the set of points;

[0010] The object is determined based on the first subset and the second subset to include a first confidence level of a first portion associated with the first subset and a second confidence level of a second portion associated with the second subset;

[0011] Whether to change the hierarchical representation of the object is determined based on the first confidence level and the second confidence level.

[0012] Secondly, this application provides an apparatus for modeling objects, comprising:

[0013] The acquisition module is used to acquire the first set of point cloud frames associated with the surrounding environment collected by the perception system;

[0014] The modeling module is used to generate a hierarchical representation of objects in the surrounding environment based on the first set of point cloud frames, wherein the hierarchical representation includes a first representation of the objects;

[0015] The acquisition module is also used to acquire a second set of point cloud frames associated with the surrounding environment collected by the perception system;

[0016] The first determining module is used to determine a set of points associated with the object in each frame of the second set of point cloud frames, as well as a first subset and a second subset of the set of points.

[0017] A confidence module is configured to determine, based on a first subset and a second subset, that the object includes a first portion associated with the first subset and a second portion associated with the second subset;

[0018] The second determining module is used to determine whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level.

[0019] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0020] The memory stores program instructions;

[0021] The processor executes program instructions stored in the memory to implement the method as described in the first aspect.

[0022] Fourthly, this application provides a computer-readable storage medium storing program instructions that, when executed by a processor, are used to implement the method described in the first aspect.

[0023] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0024] Sixthly, this application provides a method for controlling a vehicle, comprising:

[0025] Acquire the first set of point cloud frames associated with the surrounding environment collected by the sensing system;

[0026] Generate a hierarchical representation of objects in the surrounding environment based on the first set of point cloud frames. The hierarchical representation includes a first representation of the objects.

[0027] Acquire a second set of point cloud frames associated with the surrounding environment, collected by the sensing system;

[0028] Determine a set of points associated with the object in each frame of the second set of point cloud frames, as well as a first subset and a second subset of the set of points;

[0029] The object is determined based on the first subset and the second subset to include a first confidence level of a first portion associated with the first subset and a second confidence level of a second portion associated with the second subset;

[0030] Determine whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level;

[0031] Generate the path the vehicle will take based on the hierarchical representation of the object;

[0032] Control the vehicle to travel along the generated path.

[0033] This application provides a method, device, storage medium, and vehicle control method for object modeling. The method involves: acquiring a first set of point cloud frames associated with the surrounding environment from a perception system; generating a hierarchical representation of an object in the surrounding environment based on the first set of point cloud frames, the hierarchical representation including a first representation of the object; acquiring a second set of point cloud frames associated with the surrounding environment from the perception system; determining a set of points associated with the object in each frame of the second set of point cloud frames, as well as a first subset and a second subset of these points; determining, based on the first subset and the second subset, that the object includes a first confidence level of a first part associated with the first subset and a second confidence level of a second part associated with the second subset; and determining, based on the first confidence level and the second confidence level, whether to change the hierarchical representation of the object. This application can first generate a hierarchical representation of the object based on the first set of point cloud frames, and then, based on the set of points associated with the object in the subsequently acquired second set of point cloud frames, as well as the first and second subsets of these points and the corresponding confidence levels, determine whether to adjust or refine the hierarchical representation of the object. This achieves dynamic optimization of the object's representation based on observation, making the object's representation more accurate and stable with observation, thereby accurately determining the object's motion state. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] Figure 1a This application illustrates an example scenario;

[0036] Figure 1b This is a schematic diagram of a vehicle according to an example embodiment of this application;

[0037] Figure 2A flowchart of a method for modeling an object provided in one embodiment of this application;

[0038] Figure 3 A flowchart of a method for modeling an object provided in another embodiment of this application;

[0039] Figure 4a The hierarchical representation of objects provided in the embodiments of this application;

[0040] Figure 4b A tree diagram representing the hierarchy of objects according to embodiments of this application;

[0041] Figure 5 This is for implementation according to the example embodiments of this application. Figure 2 An example procedure for one step;

[0042] Figure 6 This is for implementation according to the example embodiments of this application. Figure 2 Another example of a step in the process;

[0043] Figure 7 A flowchart of a method for controlling a vehicle provided in an embodiment of this application;

[0044] Figure 8 This is a schematic diagram of a target detection method based on BEV provided in an embodiment of this application;

[0045] Figure 9 This is a schematic diagram of a target detection method based on a view frustum provided in an embodiment of this application;

[0046] Figure 10 A structural diagram of an apparatus for modeling an object provided in an embodiment of this application;

[0047] Figure 11 This is a structural diagram of an electronic device provided in an embodiment of this application.

[0048] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] Autonomous vehicles may include multiple sensors to capture sensor data corresponding to the vehicle's surrounding environment. The environment typically includes multiple objects, which can be represented using bounding boxes. For example, in a two-dimensional representation parallel to the ground, the area and shape of an object in the environment can be approximated by a rectangular bounding box, thus representing the object in the environment. In the prior art, a single bounding box is typically used to represent one object in the environment.

[0051] Objects in the environment can be articulated objects, which consist of at least two interconnected parts that can rotate relative to each other. An example of an articulated object is a truck comprising a tractor and a trailer. The inventors of this application have noted that representing an articulated object using a single bounding box (e.g., a single rectangular bounding box) fails to accurately capture the object's motion, leading to inaccurate estimations of its motion. For instance, when a truck turns, an angle is formed between the tractor and trailer. Representing the truck with a single bounding box results in a bounding box area that does not correspond to the truck's actual footprint, potentially leading to an overestimation of the truck's footprint. In particular, even when the truck actually occupies only one lane, its bounding box might occupy two lanes on the map.

[0052] To address the mismatch between the bounding box region and the actual footprint of an articulated object, each part of the articulated object can be treated as an independent object and represented by a bounding box. For example, the tractor unit and trailer of a truck can be considered as separate objects and represented by different bounding boxes. This representation is simpler, but it cannot accurately express the motion laws of the articulated object and increases the difficulty of estimating its motion state (e.g., trajectory). For instance, the tractor unit of a truck is powered, while the trailer is not, and the trailer's motion is constrained by the tractor unit's motion. Using an incorrect motion model will cause the predicted trajectories of the tractor unit and trailer to deviate significantly from each other.

[0053] Furthermore, the applicant noted that the relative positional relationships (e.g., distance and relative orientation) between the autonomous vehicle and objects in the environment change over time, and the observations of these objects by the sensors on the autonomous vehicle also change accordingly. Therefore, the modeling of objects in the environment should take into account these changes in the relative positional relationships between the autonomous vehicle and the objects in the environment.

[0054] The technology described in this application can be used to process sensor data and dynamically generate hierarchical representations of objects in an environment to accurately represent the objects, thereby navigating vehicles (such as autonomous vehicles) in the environment and avoiding relevant objects in the environment (such as other vehicles in the environment).

[0055] In summary, this application provides a method for modeling objects (i.e., a method for dynamically generating hierarchical representations of objects). This method involves: acquiring a first set of point cloud frames associated with the surrounding environment from a perception system; generating a hierarchical representation of objects in the surrounding environment based on the first set of point cloud frames, the hierarchical representation including a first representation of the object; acquiring a second set of point cloud frames associated with the surrounding environment from the perception system; determining a set of points associated with the object in each frame of the second set of point cloud frames, as well as a first subset and a second subset of these points; determining, based on the first subset and the second subset, a first confidence level for the object including a first part associated with the first subset and a second confidence level for a second part associated with the second subset; and determining whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level. This application can first generate a representation of the object based on the first set of point cloud frames, and then determine whether to adjust or refine the object representation based on a set of points associated with the object in the subsequently acquired second set of point cloud frames, as well as the first and second subsets of these points and the corresponding confidence levels. This achieves dynamic optimization of the object representation based on observations, making the object representation more accurate and stable with each observation, thereby accurately determining the object's motion state.

[0056] The object modeling method provided in this application (i.e., the method for dynamically generating hierarchical representations of objects) can be used to control a vehicle. The vehicle's computing device generates the path the vehicle will travel based on the hierarchical representation of the objects and controls the vehicle to travel along the generated path.

[0057] Figure 1a Example scenarios are shown at different times. For example... Figure 1a As shown, at the first time point, t1, vehicle 100 is traveling in one lane of the road, with truck 116 traveling in front of it. In another nearby lane, truck 112 and car 114 are traveling. At this time, vehicle 100 is relatively far from truck 112, and the field of view of vehicle 100's sensors is partially obstructed by truck 116, preventing vehicle 100's sensors from clearly observing vehicle 112. At the second time point, t2, vehicle 100 is closer to truck 112 than at time t1, allowing vehicle 100's sensors to observe vehicle 112 more clearly than at time t1. At the third time point, t3, vehicle 100 is even closer to truck 112 than at time t2, and there are no other vehicles obstructing its view, allowing vehicle 100's sensors to observe vehicle 112 more clearly than at time t2.

[0058] Vehicle 100 can be an autonomous vehicle. When planning its driving route, vehicle 100 needs to consider the motion states of trucks 112 and 116, as well as car 114. Furthermore, Figure 1a Truck 112 is shown to include a tractor and two trailers, and truck 116 includes a tractor and a trailer.

[0059] Figure 1b A vehicle is shown according to an example embodiment of this application (e.g., Figure 1a A schematic diagram of vehicle 100 shown. Figure 1b As shown, vehicle 100 includes a perception system 101 and a computing device 102. The perception system 101 may include one or more of a plurality of sensors, such as an inertial measurement unit (IMU), a global navigation satellite system (GNSS) transceiver (e.g., a global positioning system (GPS) transceiver), a radio detection and ranging (RADAR) system, a laser detection and ranging (LIDAR) system, an acoustic sensor, an ultrasonic sensor, and a camera.

[0060] The perception system 101 acquires sensor data associated with the environment surrounding the vehicle 100 (i.e., the environment surrounding the perception system 101). For example, the LiDAR in the perception system 101 can acquire point clouds associated with the environment surrounding the vehicle 100, and the camera in the perception system 101 can acquire images associated with the environment surrounding the vehicle 100. Furthermore, the LiDAR and the camera in the perception system 101 can have overlapping fields of view, so that at the same time or almost at the same time, the image acquired by the camera and the point cloud acquired by the LiDAR have data about the same object.

[0061] The perception system 101 can transmit this sensor data (e.g., point cloud frames) to the computing device 102, which can detect or identify content associated with objects in the environment surrounding the vehicle 100 from the sensor data. For example, the computing device 102 can detect or identify objects associated with the environment surrounding the vehicle 100 from the point cloud acquired by LiDAR (e.g., point cloud frames). Figure 1a The points associated with the truck 112 or 116 or the car 114. For objects such as articulated objects, the computing device 102 detects or identifies the points corresponding to the object as a whole and its individual parts from the point cloud, and then generates corresponding representations for the object as a whole and its individual parts, thereby generating a hierarchical representation of the object. An example of the representation of the object (i.e., the representation of the object as a whole) and the representations of the individual parts of the object is a bounding box, such as a rectangular bounding box.

[0062] For example, see Figure 1aAt time t1, vehicle 100 is far from truck 112. If computing device 102 detects or identifies truck 112 from the point cloud frames (i.e., the first set of point cloud frames) acquired by perception system 101 at time t1, it generates a hierarchical representation 103a of truck 112. Since vehicle 100 is far from truck 112 at this time, computing device 102 cannot identify the details of truck 112 from the first set of point cloud frames (e.g., it cannot determine whether the truck includes a trailer). Therefore, the generated hierarchical representation 103a of the truck only includes the representation corresponding to the entire truck 112 (hereinafter referred to as the first representation), that is, it only includes a bounding box. Figure 1a As shown, at time t2, relative to time t1, vehicle 100 is closer to truck 112. If computing device 102 detects or identifies the truck's tractor and trailer (i.e., the first and second parts of the truck) from the point cloud frames (i.e., the second set of point cloud frames) collected by sensing system 101 at time t2, it adjusts or changes the hierarchical representation 103a to obtain the hierarchical representation 103b of truck 112. Since vehicle 100 is closer to truck 112 at this time, computing device 102 can determine from the second set of point cloud frames that truck 112 includes not only the tractor but also the trailer, but cannot determine that the truck includes multiple trailers. Therefore, the obtained hierarchical representation 103b of the truck includes the first representation corresponding to the entire truck, the second representation corresponding to the truck's tractor, and the third representation corresponding to the truck's trailer. Figure 1a As shown, at time t3, relative to time t2, vehicle 100 is very close to truck 112 and there is no obstruction from truck 116. If computing device 102 detects or identifies not only the tractor unit of the truck but also the two trailers (i.e., the third and fourth parts of the truck) from the point cloud frames (i.e., the third set of point cloud frames) collected by perception system 101 at time t3, then the hierarchical representation 103b is adjusted or changed to obtain the hierarchical representation 103c of truck 112. Since vehicle 100 is close to truck 112 at this time, computing device 102 can clearly determine from the third set of point cloud frames that truck 112 includes two trailers. Therefore, the obtained hierarchical representation 103c of the truck includes a first representation corresponding to the entire truck, a second representation corresponding to the tractor unit of the truck, a third representation corresponding to the trailers (i.e., the first and second trailers together), a fourth representation corresponding to the first trailer, and a fifth representation corresponding to the second trailer. This achieves accurate modeling of truck 112. Based on an accurate model of the truck 112, the computing device 102 can accurately determine the motion state of the truck 112.

[0063] Figure 2 This is a flowchart illustrating a method for object modeling provided in an embodiment of this application. The method for object modeling provided in this embodiment is executed by any electronic device, such as... Figure 1b The computing device 102 shown.

[0064] like Figure 2 As shown, in step S201, the first set of point cloud frames associated with the surrounding environment is acquired by the sensing system. The first set of point cloud frames includes one or more point cloud frames.

[0065] Vehicles (e.g., Figure 1a and Figure 1b The vehicle 100 shown can employ a sensing system (e.g., when in motion) Figure 1b The perception system 101 shown acquires sensor data associated with the environment surrounding the vehicle 100 (i.e., the environment surrounding the perception system 101). The sensor data acquired by the perception system includes, for example, point clouds acquired by the LiDAR in the perception system and associated with the environment surrounding the vehicle 100, and images acquired by the camera in the perception system and associated with the environment surrounding the vehicle 100. The perception system 101 can send one or more frames of point clouds acquired by the LiDAR (e.g., a first set of point cloud frames) to the computing device 102. In addition to point clouds, the perception system 101 can also send one or more frames of images acquired by the camera or data acquired by other sensors to the computing device 102.

[0066] Some objects in the environment (e.g.) Figure 1a Trucks 112 and 116 may be articulated objects, while other objects may be non-articulated objects (e.g., Figure 1a (e.g., a passenger car 114). An articulated object comprises at least two parts that can rotate relative to each other. For example, the articulated object is a truck, which includes a tractor unit and a trailer, the trailer being rotatable relative to a specific point on the tractor unit. When a truck includes multiple trailers, one trailer is rotatable relative to a specific point on another trailer.

[0067] In step S202, a hierarchical representation of objects in the surrounding environment is generated based on the first set of point cloud frames. The hierarchical representation includes a first representation of the objects.

[0068] After receiving sensor data including a first set of point cloud frames from the perception system 101, the computing device 102 can use various target detection methods to perform target detection on the first set of point cloud frames, and can determine objects in the vehicle's surrounding environment (e.g., objects in the first set of point cloud frames) from the first set of point cloud frames. Figure 1a A set of points associated with truck 112.

[0069] When the perception system 101 acquires the first set of point cloud frames, if the vehicle 100 is far from the object or the object is partially occluded, the sensors in the perception system 101 may not be able to clearly observe the object, or may only be able to observe a part of the object. At this time, the computing device 102 cannot determine whether the object includes at least two relatively rotatable parts based on the first set of point cloud frames; the overall representation of the object (i.e., the first representation of the object) constitutes the hierarchical representation of the object. In other words, at this time, the hierarchical representation of the object only includes the first representation of the object. For example, in the perception system 101... Figure 1a When the first set of point cloud frames is acquired at time t1, the sensors of vehicle 100 may only observe the tractor unit of truck 112, but not the trailer unit. Computing device 102 will determine a set of points associated with truck 112 and generate a representation of truck 112 as a whole based on this set of points. At this point, the representation of truck 112 as a whole constitutes the hierarchical representation of truck 112.

[0070] An example of the first representation of an object is a bounding box, such as a rectangular bounding box.

[0071] In step S203, the second set of point cloud frames associated with the surrounding environment collected by the sensing system is acquired.

[0072] In this embodiment, the second set of point cloud frames associated with the surrounding environment is one or more point cloud frames acquired by the perception system 101 after acquiring the first set of point cloud frames. As time goes by, the distance between the perception system 101 and the object may become closer, or the object may no longer be occluded, allowing the perception system 101 to observe more parts or details of the object. Therefore, the second set of point cloud frames can further reflect more parts or details of the object in more detail, and under certain conditions, the hierarchical representation of the object can be refined based on the second set of point cloud frames.

[0073] For example, the sensing system 101 in Figure 1a The first set of point cloud frames was acquired at time t1. Figure 1a The second set of point cloud frames is acquired after time t1 and before time t2 (or at time t2). The closer the second set of point cloud frames is acquired by the perception system 101 to time t2, the closer the vehicle 100 is to the truck 112, and the clearer the observation of the truck 112 by the sensors of the vehicle 100.

[0074] In step S204, a set of points associated with the object in each frame of the second set of point cloud frames is determined, along with a first subset and a second subset of the set of points.

[0075] After receiving sensor data including a second set of point cloud frames from the perception system 101, the computing device 102 can determine a set of points associated with an object in the vehicle's surrounding environment (the same object detected from the first set of point cloud frames) from each frame of the second set of point cloud frames, and determine multiple subsets of the set of points, such as a first subset and a second subset, and generate a representation (e.g., a bounding box) of one or more parts of the object based on the subsets of the set of points.

[0076] The computing device 102 can use various target detection methods to perform target detection on the second set of point cloud frames, and determine a set of points associated with the object in each frame of the second set of point cloud frames, as well as a first subset and a second subset of the set of points.

[0077] In step S205, the object is determined to include a first confidence level of a first part associated with the first subset and a second confidence level of a second part associated with the second subset based on the first subset and the second subset.

[0078] The computing device 102 determines a set of points in each frame of the second set of point cloud frames that are associated with an environmental object, and determines a subset of this set of points (e.g., a first subset and a second subset). Simultaneously or subsequently, the computing device 102 can determine a first confidence level corresponding to the first subset and a second confidence level corresponding to the second subset based on at least some or all of the first subset and at least some or all of the second subset.

[0079] In step S206, determine whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level.

[0080] The computing device 102 determines whether to change the hierarchical representation of the object based on a first confidence level corresponding to a first subset (i.e., the confidence level that the object includes a first part associated with the first subset) and a second confidence level corresponding to a second subset (i.e., the confidence level that the object includes a second part associated with the second subset). For example, the computing device 102 determines whether to change the hierarchical representation of the object by judging the relationship between the first confidence level, the second confidence level, and a predetermined threshold. The predetermined threshold can be determined empirically. If both the first confidence level and the second confidence level are low (e.g., below the predetermined threshold), it indicates that the detection result is inaccurate, and the detection result is discarded; if at least one of the first confidence level and the second confidence level is higher than the predetermined threshold, the hierarchical representation of the object is changed.

[0081] In some embodiments, in response to both the first confidence level and the second confidence level being lower than a predetermined threshold, the hierarchical representation of the object remains unchanged. That is, when it is determined that both the first confidence level and the second confidence level are lower than the predetermined threshold, it indicates that the detection result is inaccurate, that is, the first subset and the second subset determined from a set of points associated with the object are inaccurate. Therefore, the detection result is discarded, and the representation of the first part of the object and the representation of the second part of the object are not generated based on the first subset and the second subset. Instead, the hierarchical representation of the object generated in S202 is still used.

[0082] In some embodiments, in response to one of the first confidence level and the second confidence level being lower than a predetermined threshold and the other being higher than a predetermined threshold, the representation of the object is adjusted (e.g., at least one of the size and shape of the first representation of the object generated in step S202 is changed). For example, upon determining that the first confidence level is higher than a predetermined threshold and the second confidence level is lower than a predetermined threshold, a representation of the first portion of the object is generated based on a first subset, while simultaneously changing at least one of the size and shape of the first representation generated in step S202 to make it the same as the representation of the first portion of the object.

[0083] For example, if the first part associated with the first subset is a truck tractor unit, and the second part associated with the second subset is a truck trailer, and the first confidence level corresponding to the first subset is higher than a predetermined threshold, while the second confidence level corresponding to the second subset is lower than the predetermined threshold, this indicates that the tractor unit is not connected to the trailer, or the trailer is obscured. The computing device 102 can change the hierarchical representation of the object generated in step S202, that is, adjust the first representation generated in step S202 to be the same as the representation of the tractor unit generated based on the first subset.

[0084] In some embodiments, in response to both the first confidence level and the second confidence level being higher than a predetermined threshold, a second representation of a first portion of the object and a third representation of a second portion of the object are generated based on a first subset and a second subset. In some embodiments, in response to both the first confidence level and the second confidence level being higher than the predetermined threshold, the computing device 102 further adjusts the first representation of the object generated in step S202 based on a set of points associated with the environmental object in a second set of point cloud frames (e.g., changing at least one of the size and shape of the first representation of the object generated in step S202).

[0085] When both the first confidence level and the second confidence level are determined to be higher than a predetermined threshold, it indicates that the perception system 101 clearly observes the articulated object, and that the perception system 101 clearly observes at least two parts of the articulated object. Therefore, a second representation of the first part of the object and a third representation of the second part of the object can be generated based on the first subset and the second subset. Simultaneously, the hierarchical representation of the object generated in step S202 is modified to further include the second representation of the first part and the third representation of the second part. Optionally, the second representation of the first part of the object and the third representation of the second part of the object are at least partially located within the first representation of the object. An example of the first representation of the object and the representations of the various parts of the object (i.e., the second and third representations) is a bounding box, such as a rectangular bounding box.

[0086] by Figure 1a For example, if the second set of point cloud frames is the result of perception system 101 in... Figure 1a The data collected at time t2, the first part associated with the first subset and the second part associated with the second subset, correspond to the tractor and trailer of truck 112, respectively. Since the perception system 101 can clearly observe the tractor and trailer at this time, the first confidence level and the second confidence level are both higher than the predetermined threshold.

[0087] Sensing system 101 in Figure 1a Between time t1 and t2, the observation of truck 112 gradually becomes clearer. Therefore, if the second set of point cloud frames is the result of the perception system 101 in... Figure 1a If the data is collected between time t1 and time t2, then the closer the acquisition time of the second set of point cloud frames is to t2, the greater the first and second confidence levels; the closer it is to t1, the smaller the first and second confidence levels.

[0088] Figure 3 A flowchart is shown for a method for modeling objects (i.e., a method for dynamically generating hierarchical representations of objects) according to another embodiment of this application. Figure 3 The method shown can be executed by any electronic device, such as... Figure 1b The computing device 102 shown. Relative to... Figure 2 The method Figure 3 This method allows for more detailed modeling of objects. Figure 3 Steps S301-S306 in the process Figure 2 Steps S201-S206 are the same, therefore, these steps will not be described in detail below.

[0089] like Figure 3 As shown, in step S307, the third set of point cloud frames associated with the surrounding environment collected by the sensing system is acquired.

[0090] The third set of point cloud frames, associated with the surrounding environment, is one or more point cloud frames acquired by the perception system 101 after acquiring the second set of point cloud frames. As time progresses, the distance between the perception system 101 and the object may become closer, or the object may no longer be occluded, allowing the perception system 101 to observe more parts or details of the object. Therefore, the third set of point cloud frames can further reflect more parts or details of the object in greater detail, and under certain conditions, the hierarchical representation of the object can be refined based on the third set of point cloud frames.

[0091] For example, the sensing system 101 in Figure 1a The second set of point cloud frames was acquired at time t2. Figure 1a The third set of point cloud frames is acquired after time t2 and before time t3 (or at time t3). The closer the third set of point cloud frames is acquired by the perception system 101 to time t3, the closer the vehicle 100 is to the truck 112, and the clearer the observation of the truck 112 by the sensors of the vehicle 100.

[0092] In step S308, a set of points in each frame of the third set of point cloud frames that are associated with the second part of the object, as well as a third subset and a fourth subset of the set of points, are determined.

[0093] After receiving sensor data including a third set of point cloud frames from the perception system 101, the computing device 102 can determine from the third set of point cloud frames each part (e.g., the second part, such as...) of an object in the vehicle's surrounding environment (which is the same object detected from the first and second set of point cloud frames). Figure 1a The system identifies a set of points associated with the trailers of truck 110 and determines multiple subsets of that set of points, such as a third subset and a fourth subset (e.g., the first trailer and the second trailer of the truck), and generates a representation (e.g., a bounding box) of one or more parts of the second part of the object based on these subsets of points.

[0094] The computing device 102 can use various target detection methods to perform target detection on the third set of point cloud frames, and determine a set of points in each frame of the third set of point cloud frames that are associated with the second part of the object, as well as a subset of that set of points (i.e., the third subset and the fourth subset).

[0095] In step S309, the second part is determined to include a third confidence level of the third part associated with the third subset and a fourth confidence level of the fourth part associated with the fourth subset based on the third subset and the fourth subset.

[0096] The computing device 102 determines a set of points in the third set of point cloud frames that are associated with the second part of the environment object, and determines a subset of this set of points (e.g., the third subset and the fourth subset). Simultaneously or subsequently, the computing device 102 can simultaneously determine the confidence levels of the third subset and the fourth subset (i.e., the third confidence level corresponding to the third subset and the fourth confidence level corresponding to the fourth subset).

[0097] In step S310, it is determined whether to further change the hierarchical representation of the object based on the third confidence level and the fourth confidence level. The computing device 102 determines whether to further change the hierarchical representation of the object based on the third confidence level corresponding to the third subset (i.e., the confidence level that the second part of the object includes the third part associated with the third subset) and the fourth confidence level corresponding to the fourth subset (i.e., the confidence level that the second part of the object includes the fourth part associated with the fourth subset). For example, the computing device 102 determines whether to change the hierarchical representation of the object by judging the relationship between the third confidence level and the fourth confidence level and a predetermined threshold. If both the third confidence level and the fourth confidence level are low (e.g., below the predetermined threshold), it indicates that the detection result is inaccurate, and the detection result is discarded; if it is determined that at least one of the third confidence level and the fourth confidence level is higher than the predetermined threshold, the hierarchical representation of the object is changed.

[0098] In some embodiments, in response to both the third and fourth confidence levels being higher than a predetermined threshold, a fourth representation of the third part of the object and a fifth representation of the fourth part of the object are generated based on the third and fourth subsets. Simultaneously, the hierarchical representation of the object is modified to further include the fourth and fifth representations of the third and fourth parts. In some embodiments, in response to both the third and fourth confidence levels being higher than a predetermined threshold, the computing device 102 adjusts at least one of the first representation of the object, the second representation of the first part of the object, and the third representation of the second part of the object based on a third set of point cloud frames.

[0099] When both the third and fourth confidence levels are determined to be higher than a predetermined threshold, it indicates that the perception system 101 clearly observes that the articulated object comprises at least three parts. Figure 4b This illustrates the hierarchical representation of the object at this point, including a representation of the object as a whole, and representations of the observed parts of the object (the first part of the object, the second part of the object, the third part of the object, and the fourth part of the object). Optionally, the fourth representation of the third part included in the second part of the object, and the fifth representation of the fourth part included in the second part of the object, are at least partially located within the third representation of the second part of the object.

[0100] by Figure 1a For example, if the third set of point cloud frames is the sensor system 101 in Figure 1aThe data collected at time t3, the third part associated with the third subset and the fourth part associated with the fourth subset, correspond to different trailers of truck 112 respectively. Since at this time, the perception system 101 can clearly observe each trailer, the third confidence level and the fourth confidence level are both higher than the predetermined threshold.

[0101] Sensing system 101 in Figure 1a Between time t2 and t3, the observation of truck 112 gradually becomes clearer. Therefore, if the third set of point cloud frames is from the sensing system 101... Figure 1a If the data was collected between time t2 and time t3, then the closer the acquisition time of the third set of point cloud frames is to time t3, the greater the third and fourth confidence scores; the closer it is to time t2, the smaller the third and fourth confidence scores.

[0102] Optionally, in response to the fact that both the third and fourth confidence levels are below a predetermined threshold, indicating that the detection result is inaccurate, the detection result is discarded, and the hierarchical representation of the object remains unchanged.

[0103] Optionally, in response to one of the third and fourth confidence levels being below a predetermined threshold and the other being above a predetermined threshold, the representation of the object is adjusted (e.g., at least one of the size and shape of the object's third representation is changed). For example, upon determining that the third confidence level is above a predetermined threshold and the fourth confidence level is below a predetermined threshold, a fourth representation of the object's third portion is generated based on the third subset, while simultaneously changing at least one of the size and shape of the third representation to make it identical to the fourth representation of the object's third portion.

[0104] It should be noted that if the second part of an object includes not only the third and fourth parts, but also more parts, the remaining parts can also be generated using the same process described above, which will not be elaborated further here. Furthermore, if the first part of an object may also include different parts, the same process described above can also be used to generate the corresponding representation.

[0105] This embodiment provides a method for modeling objects by acquiring a first set of point cloud frames associated with the surrounding environment collected by a perception system; generating a hierarchical representation of the object in the surrounding environment based on the first set of point cloud frames, the hierarchical representation including a first representation of the object; acquiring a second set of point cloud frames associated with the surrounding environment collected by the perception system; determining a set of points associated with the object in each frame of the second set of point cloud frames, as well as a first subset and a second subset of these points; determining, based on the first subset and the second subset, that the object includes a first confidence level of a first part associated with the first subset and a second confidence level of a second part associated with the second subset; and determining, based on the first confidence level and the second confidence level, whether to change the hierarchical representation of the object. This embodiment can first generate a hierarchical representation of the object based on the first set of point cloud frames, and then, based on the set of points associated with the object in the subsequently acquired second set of point cloud frames, as well as the first and second subsets of these points and the corresponding confidence levels, determine whether to adjust or refine the hierarchical representation of the object. This achieves dynamic optimization of the object's representation based on observations, making the object's representation more accurate and stable with observations, thereby accurately determining the object's motion state.

[0106] Figure 4a This illustrates a hierarchical representation of an object (e.g., an articulated object, such as a truck) generated using sensor data acquired by a perception system (e.g., perception system 101 of vehicle 100) according to an example embodiment of this application (see above). Figure 1b The described hierarchy represents 103c or reference Figure 3 An example of the generated hierarchical representation.

[0107] like Figure 4a As shown, the truck hierarchy representation 410 includes a bounding box 411. The bounding box 411 is determined based on a set of points associated with the truck in the first set of point cloud frames. The shape and size of the bounding box 411 correspond to the overall shape and size of the truck. That is, the shape and size of the bounding box 411 approximate the shape and size of the truck.

[0108] The truck hierarchy representation also includes a bounding box 412. Bounding box 412 is determined based on a subset of points associated with the truck in a second set of point cloud frames at another time (i.e., the set of points associated with the tractor unit in a second set of point cloud frames at another time). The shape and size of bounding box 412 correspond to the shape and size of the tractor unit. In other words, the shape and size of bounding box 412 approximate the shape and size of the tractor unit.

[0109] The truck hierarchy representation also includes a bounding box 413. Bounding box 413 is determined based on a subset of the points associated with the truck in the second set of point cloud frames (i.e., the set of points associated with the trailer in the second set of point cloud frames). The shape and size of bounding box 413 correspond to the shape and size of the trailer. In other words, the shape and size of bounding box 413 approximate the shape and size of the trailer. Figure 4a In the middle, bounding boxes 412 and 413 are completely located within bounding box 411.

[0110] When a truck includes multiple trailers (e.g., trailer 1 and trailer 2), the truck hierarchy representation also includes bounding boxes 414 and 415. Bounding boxes 414 and 415 are determined based on a subset of the set of points associated with the trailers in a third set of point cloud frames at a different time step (i.e., points associated with trailer 1 and points associated with trailer 2 in the third set of point cloud frames). The shapes and dimensions of bounding boxes 414 and 415 correspond to the shapes and dimensions of trailer 1 and trailer 2, respectively. In other words, the shapes and dimensions of bounding boxes 414 and 415 approximate the shapes and dimensions of trailer 1 and trailer 2. Figure 4a In the middle, bounding boxes 414 and 415 are completely located within bounding box 413.

[0111] Figure 4b This is a tree diagram 420 representing the hierarchical structure of articulated objects (such as trucks). For example... Figure 4b As shown, the first level of the tree diagram 420 includes nodes 421, which correspond to the overall representation of the truck, for example... Figure 4a The bounding box 411 in the diagram. That is, the overall representation of the truck is at the first level of the truck's hierarchical representation. The second level of the tree diagram 420 includes nodes 422 and 423, corresponding to the representation of the tractor and the overall representation of the trailer, respectively, for example... Figure 4a Bounding boxes 412 and 413 in the diagram. That is, the representation of the tractor and the overall representation of the trailer are at the second level of the truck's hierarchical representation. Nodes 424 and 425 at the third level of the tree diagram 420 correspond to the representations of different trailers (e.g., trailer 1 and trailer 2), for example... Figure 4a Bounding boxes 414 and 415 are shown in the diagram. That is, the representations of trailer 1 and trailer 2 are at the third level of the truck's hierarchical representation. If a truck includes a tractor and two trailers, such as trailer 1 and trailer 2, located near vehicle 100, and the vehicle 100's perception system 101 can observe the tractor and two trailers, the vehicle 100's computing device 102 generates the following hierarchical representation for the truck: Figure 4bAs shown, it includes three levels. When the vehicle's perception system 101 cannot fully observe the truck, for example, only the truck's tractor and trailer are observed, the computing device 102 of the vehicle 100 generates a hierarchical representation for the truck, including two levels: the first level and the second level. When the vehicle's perception system 101 cannot fully observe the truck, for example, only the truck's tractor is observed, the computing device 102 of the vehicle 100 generates a hierarchical representation for the truck, including only one level: the first level.

[0112] Figure 5 This is a process according to an example embodiment of this application, used to implement Figure 2 Step S205 (and Figure 3 Step S305): Determine, based on the first subset and the second subset, that the object includes a first confidence level of a first part associated with the first subset and a second confidence level of a second part associated with the second subset. Figure 5 The process shown is executed by any electronic device, such as... Figure 1b The computing device 102 shown.

[0113] like Figure 5 As shown, in step 501, for each point cloud frame of the second set of point cloud frames, a first probability is determined based on the first subset that the object includes the first part.

[0114] In step 502, for each point cloud frame of the second set of point cloud frames, a second probability is determined based on the second subset that the object includes the second part;

[0115] In step 503, the first probability of the multi-frame point cloud of the second group of point cloud frames is accumulated, and the second probability of the multi-frame point cloud of the second group of point cloud frames is accumulated.

[0116] For each point cloud frame in the second set of point cloud frames, the probability that the object includes the first part is determined, denoted as the first probability. The first probability corresponding to each point cloud frame in the second set of point cloud frames can be accumulated, and the result can be determined as the first confidence level that the object includes the first part associated with the first subset. Similarly, for each point cloud frame in the second set of point cloud frames, the probability that the object includes the second part is determined, denoted as the second probability. The second probability corresponding to each point cloud frame in the second set of point cloud frames can be accumulated, and the result can be determined as the second confidence level that the object includes the second part associated with the second subset.

[0117] Figure 6 This is a process according to another example embodiment of the present application, used to implement Figure 2 Step S205 (and Figure 3Step S305): Determine, based on the first subset and the second subset, that the object includes a first confidence level of a first part associated with the first subset and a second confidence level of a second part associated with the second subset. Figure 6 The process shown is executed by any electronic device, such as... Figure 1b The computing device 102 shown. Figure 6 Steps S601-S603 and Figure 5 Steps S501-S503 are the same, therefore, these steps will not be described in detail below.

[0118] like Figure 6 As shown, in step 604, for each point cloud frame of the second set of point cloud frames, a first association probability that the object includes the first part is determined based on the first subset and the second subset.

[0119] In step 605, for each point cloud frame of the second set of point cloud frames, a second association probability is determined based on the first subset and the second subset to indicate that the object includes the second part.

[0120] In step 606, the first association probability of the multi-frame point cloud of the second group of point cloud frames is accumulated, and the second association probability of the multi-frame point cloud of the second group of point cloud frames is accumulated.

[0121] When a set of points associated with an object includes a first subset and a second subset, any point in the set of points associated with the object belongs to either the first subset or the second subset. That is, the probability that any point in the set of points associated with the object belongs to the first subset is influenced by both the first and second subsets, and the probability that any point in the set of points associated with the object belongs to the second subset is influenced by both the second and first subsets. Therefore, for each point cloud frame of the second set of point cloud frames, a first association probability that the object includes the first part can be determined based on the first and second subsets. The first association probability corresponding to each point cloud frame of the second set of point cloud frames can be accumulated, and multiplied by the accumulated first probability to obtain the result, which is determined as the first confidence level that the object includes the first part associated with the first subset. Similarly, a second association probability that the object includes the second part can be determined based on the first and second subsets. The second association probability corresponding to each point cloud frame of the second set of point cloud frames can be accumulated, and multiplied by the accumulated second probability to obtain the result, which is determined as the second confidence level that the object includes the second part associated with the second subset.

[0122] In one alternative embodiment, the first association probability of each point cloud in the second set of point cloud frames can be accumulated, and the result can be determined as a first confidence level that the object includes a first part associated with the first subset; the second association probability of each point cloud in the second set of point cloud frames can be accumulated, and the result can be determined as a second confidence level that the object includes a second part associated with the second subset.

[0123] In some embodiments, one or more frames of point clouds associated with a specific environment are acquired in advance using a sensor (e.g., LiDAR), and each frame of point cloud is labeled (e.g., manually) with a set of points associated with an articulated object in the environment, a first subset of the set of points corresponding to a first part of the articulated object, and a second subset of the set of points corresponding to a second part of the articulated object. These point clouds and labeling results are stored as a dataset. Then, for each frame of point cloud, the distance between the object in the environment (i.e., the articulated object) and the LiDAR at the time of acquiring the point cloud is determined, and a set of points associated with the environmental object in the dataset, as well as a first subset and a second subset of the set of points, are determined using the method described above in this application. A first probability is determined based on the first subset, and a second probability is determined based on the second subset. Then, the accuracy corresponding to each first probability and each second probability is determined by combining the labeling results. Thus, for each frame of point cloud, the calculated accuracy and the distance between the object and the LiDAR at the time of acquiring the point cloud are stored in the dataset.

[0124] When determining the accuracy of the first and second probabilities for each point cloud frame in the second set of point cloud frames, the distance between the object and the perception system can be determined first based on each point cloud frame in the second set of point cloud frames. The accuracy corresponding to the distance with the smallest difference from the target distance is then selected from the stored dataset and used as the accuracy of the first and second probabilities for that point cloud frame in the second set of point cloud frames. Accordingly, Figure 5 and Figure 6 The process may further include: before accumulating the first and second probabilities in step S503 or step S603, determining the corresponding accuracy for the first and second probabilities corresponding to each frame of the second set of point cloud frames based on the pre-stored dataset. At this time, accumulating the first and second probabilities in step S503 or step S603 includes:

[0125] For each initial probability, the corresponding accuracy is used for correction.

[0126] For each second probability, the corresponding accuracy is used for correction.

[0127] Accumulate the corrected first probability and corrected second probability corresponding to the multi-frame point clouds of the second group of point cloud frames.

[0128] When accumulating the first probability of each point cloud frame in the second set of point cloud frames, the accuracy corresponding to the first probability is also multiplied, and the result (or the result is multiplied by the accumulation of the first associated probabilities) is determined as the first confidence that the object includes the first part associated with the first subset; when accumulating the second probability of each point cloud frame in the second set of point cloud frames, the accuracy corresponding to the second probability is also multiplied, and the result (or the result is multiplied by the accumulation of the second associated probabilities) is determined as the second confidence that the object includes the second part associated with the second subset.

[0129] When considering accuracy, the confidence level of any part of an object can be determined using the following formula:

[0130]

[0131] Wherein, P(y i =T|X) is a set of point cloud frames X, under the condition that there exists an i-th part of an object (i.e., y). i =True) confidence level, where a set of point cloud frames X includes point cloud frames at t time points, Z is a normalization constant (used to set the result between 0 and 1); ∏ represents accumulation, here it is accumulation over t time points;

[0132]

[0133] in, This represents the probability that the i-th part of an object exists. This represents the probability of predicting the existence of the i-th part of an object based on point cloud frame X (which could be the confidence level of the model in recognizing the i-th part of the object). This represents the accuracy (which could be the model accuracy) of the predicted portion of the i-th object determined based on a pre-stored dataset; optionally, for First, determine the distance between the object and the perception system for each point cloud frame in the set of point cloud frames X, and select the accuracy (i.e. the accuracy of predicting the existence of the i-th part of the object) at that distance (or the distance with the smallest difference) from the pre-stored dataset.

[0134] Alternatively, considering accuracy, the confidence level for any part of the object can be determined using the following formula:

[0135]

[0136] In the formula The previous parts have already been explained above. The following section will focus on describing...

[0137]

[0138] in, This represents the probability of association between part i of an object and part j, given that both parts i and j are simultaneously identified in a point cloud frame. The association probability can be calculated from point cloud frame X. To simplify the calculation, the association probability can also be calculated from point cloud frames in a pre-stored dataset. For example, for... First, determine the distance between the object and the perception system for each point cloud frame in the set of point cloud frames X, and select the association probability (i.e., predict the association probability of the i-th part of the object when the i-th part and the j-th part of the object are identified at the same time) from the pre-stored dataset.

[0139] It should be noted that the above formula can be applied to determine the first confidence level that the object includes a first part associated with the first subset and a second confidence level that includes a second part associated with the second subset, based on the first subset and the second subset. It can also be applied to determine the confidence level that the second part includes a third part and the confidence level that includes a fourth part.

[0140] Figure 7 An example of controlling a vehicle (e.g.) is shown. Figure 1a and Figure 1b A flowchart of a method for a vehicle 100 in the process, which may be, for example, by Figure 1b The computing device 102 performs the operation.

[0141] In step S701, the computing device 102 generates a hierarchical representation of objects in the environment surrounding the vehicle 100. The computing device 102 on the vehicle 100 can generate this hierarchical representation of objects in the environment surrounding the vehicle 100 based on data (e.g., point clouds and / or images) associated with the surrounding environment collected by the perception system 101. This can be done according to any of the methods described above (e.g., Figure 2 and Figure 3 The method shown generates a hierarchical representation of the objects. Thus, in some embodiments, step S701 includes... Figure 2 In some embodiments, step S701 includes steps S201-S206. Figure 3 In steps S301-S310, the computing device 102 provides the hierarchical representation that remains unchanged or changes as in step S206 or S310 to step S702.

[0142] In step S702, computing device 102 generates the path to be traveled by the vehicle based on the hierarchical representation of the object. Computing device 102 may utilize the hierarchical representation of the object (e.g., the hierarchical representation maintained or changed in step S206 or S310) and combine it with data from perception system 101, such as data from GPS transceiver, RADAR data, LIDAR data, camera data, and / or data from other vehicle systems, to determine the driving path or trajectory of vehicle 100.

[0143] In step S703, the computing device 102 controls the vehicle to travel along the generated path. The computing device 102 can generate control commands and send them to the control system of the vehicle 100 to control the operation of the vehicle 100 and its components (or units) so that the vehicle travels along the path generated in step 702. The vehicle's control system may include various units, such as a steering unit, a power control unit, and a braking unit.

[0144] This application Figure 2 and 3 The method described can be implemented using various object detection methods, such as BEV (bird's eye view) based object detection methods and frustum based object detection methods.

[0145] The following is combined with Figure 8 This paper describes a target detection method based on BEV using the PointPillars algorithm model (hereinafter referred to as PointPillars) as an example. Those skilled in the art will understand that, in addition to PointPillars, BEV-based target detection methods can also be implemented using other neural network or algorithm models.

[0146] The traditional PointPillars network architecture includes an input network, a feature extraction network, a detection head, and an output network. Figure 8 The network architecture of PointPillars according to this application is shown. Figure 8 As shown, the PointPillars network architecture of this application, compared to traditional network architectures, additionally includes detection heads for performing Region of Interest (ROI) pooling, such as detection head 804 and detection head 805. Those skilled in the art will understand that, although... Figure 8 Two detector heads performing ROI pooling are shown, but PointPillars' network architecture can include more or fewer detector heads performing ROI pooling.

[0147] like Figure 8 As shown, the input network 801 is used to receive point clouds, for example... Figure 1bThe LiDAR sensor 101 acquires a point cloud and transmits the received point cloud to a feature extraction network 802. The feature extraction network 802 segments the point cloud received from the input network 801 into multiple pillar units, converting the point cloud into a pseudo-image, and extracts features from the pseudo-image using a convolutional neural network. The detection head 803 performs target detection based on the features extracted by the feature extraction network 802, thereby determining a set of points in the point cloud associated with objects in the environment surrounding the vehicle (e.g., vehicle 100), generating corresponding object representations (e.g., bounding boxes), and generating confidence scores (i.e., the confidence that the object exists) for that set of points. The detection head 804 then performs ROI pooling operations on this set of points, determining multiple subsets of these points (e.g., a first subset and a second subset) to predict multiple parts of the object, such as a first part associated with the first subset and a second part associated with the second subset, and generating bounding boxes for the first and second parts, as well as confidence scores corresponding to the first and second parts (i.e., the confidence that the object includes the first part and the confidence that the object includes the second part). The detection head 805 then performs ROI pooling on at least one subset (e.g., a second subset) of the plurality of subsets determined by the detection head 804, determining multiple subsets of the second subset (e.g., a third subset and a fourth subset) to predict multiple parts of the second part of the object, such as a third part associated with the third subset and a fourth part associated with the fourth subset, and generates bounding boxes for the third and fourth parts, as well as confidence scores corresponding to the third and fourth parts (i.e., confidence scores for the object including the third part and confidence scores for the object including the fourth part). The output network 806 outputs a hierarchical representation of the object composed of bounding boxes based on the aforementioned confidence scores.

[0148] In some embodiments, after performing target detection, the detection head 803 determines the category of the detected object. If the detected object is a hinged object, the detection result of the detection head 803 is provided to the detection head 804 for ROI pooling operation. If the detected object is not a hinged object, the detection result is not provided to the detection head 804, but is directly provided to the output network 806 for output.

[0149] The following is combined with Figure 9 This section describes in detail the frustum-based object detection method. The traditional frustum-based object detection method's network architecture includes an input network, a feature extraction network, a detection head, and an output network. Figure 9 The network architecture of the view frustum-based target detection method according to this application is shown. The network architecture of this application improves upon traditional network architecture detection heads by, on the one hand, reducing the detection head (e.g., ...) Figure 9 The detection head 903 is divided into multiple branches, and on the other hand, the detection head is increased (for example, by adding...). Figure 9 (Detection head 904).

[0150] As described above, the sensing system 101 (see...) Figure 1b In this context, the LIDAR and camera can have overlapping fields of view. The target detection method based on the view frustum takes advantage of the overlapping fields of view between the LIDAR and the camera, and combines the image acquired by the camera with the point cloud acquired by the LIDAR to perform target detection.

[0151] like Figure 9 As shown, the input network 901 receives point clouds and images, such as point clouds acquired by the LiDAR of the perception system 101 and images acquired by the camera of the perception system 101. The input network 901 detects the bounding boxes of objects in the image, and then extends along the direction of the bounding boxes with the camera as the origin to form a cone. The input network 901 determines which points in the point cloud are located within this cone based on the relative pose of the camera and the LiDAR.

[0152] Feature extraction network 902 is used to extract features from points within a cone determined by input network 901. An example of feature extraction network 902 is the PointNet network. Detection head 903 performs object detection based on the features extracted by feature extraction network 902. Detection head 903 includes multiple (e.g., three) branches. One branch determines a set of points in the point cloud associated with an object in the environment surrounding the vehicle (e.g., vehicle 100), generates a representation of the object (e.g., a bounding box), and generates a confidence score (i.e., confidence that the object exists) corresponding to the set of points. Other branches of detection head 903 determine multiple subsets of the set of points (e.g., a first subset and a second subset, where each branch determines one subset) to predict multiple parts of the object, such as a first part associated with the first subset and a second part associated with the second subset (where each branch predicts one part), and generates bounding boxes for the first and second parts, as well as confidence scores corresponding to the first and second parts (i.e., confidence that the object includes the first part and confidence that it includes the second part). Detection head 904, for at least one subset of the plurality of subsets determined by detection head 903, such as a second subset, determines multiple subsets of the second subset (e.g., a third subset and a fourth subset) to predict multiple parts of a second part of an object, such as a third part associated with the third subset and a fourth part associated with the fourth subset, and generates bounding boxes for the third and fourth parts, as well as corresponding confidence scores for the third and fourth parts (i.e., confidence scores for the object including the third part and confidence scores for including the fourth part). Output network 905 outputs a hierarchical representation of the object composed of bounding boxes based on the aforementioned confidence scores.

[0153] Figure 10This is a structural diagram of an apparatus for object modeling provided in an embodiment of this application. The apparatus for object modeling provided in this embodiment is installed in any electronic device and can execute the processing flow provided in the method embodiment for object modeling, such as... Figure 10 As shown, the apparatus 1000 for modeling objects includes: an acquisition module 1001, a modeling module 1002, a determination module 1003, and a confidence module 1004.

[0154] The acquisition module 1001 is used to acquire the first set of point cloud frames associated with the surrounding environment collected by the perception system.

[0155] The modeling module 1002 is used to generate a hierarchical representation of objects in the surrounding environment based on the first set of point cloud frames. The hierarchical representation includes a first table of objects.

[0156] The acquisition module 1001 is further configured to acquire a second set of point cloud frames associated with the surrounding environment collected by the perception system.

[0157] The first determining module 1003 is used to determine a set of points associated with the object in each frame of the second set of point cloud frames, as well as a first subset and a second subset of the set of points.

[0158] The confidence module 1004 is configured to determine, based on a first subset and a second subset, that the object includes a first confidence level of a first portion associated with the first subset and a second confidence level of a second portion associated with the second subset.

[0159] The second determining module 1005 is used to determine whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level.

[0160] In one or more embodiments of this application, the modeling module 1002 is further configured to:

[0161] In response to both the first confidence level and the second confidence level being lower than a predetermined threshold, the hierarchical representation remains unchanged.

[0162] In one or more embodiments of this application, the modeling module 1002 is further configured to:

[0163] In response to at least one of the first confidence level and the second confidence level being higher than a predetermined threshold, the hierarchical representation is changed.

[0164] In one or more embodiments of this application, the modeling module 1002 is further configured to:

[0165] In response to one of the first confidence level and the second confidence level being lower than a predetermined threshold and the other being higher than a predetermined threshold, at least one of the size and shape of the first representation is changed.

[0166] In response to both the first confidence level and the second confidence level being higher than a predetermined threshold, a second representation of the first part of the object and a third representation of the second part of the object are generated.

[0167] The hierarchical representation is modified to further include a second representation of the first portion and a third representation of the second portion, wherein the second and third representations are at least partially located within the first representation.

[0168] In one or more embodiments of this application, when the confidence module 1004 determines, based on a first subset and a second subset, that the object includes a first portion associated with the first subset and a second portion associated with the second subset, it is configured to:

[0169] For each point cloud frame in the second set of point cloud frames, a first probability is determined based on the first subset that the object includes the first part.

[0170] For each point cloud frame in the second set of point cloud frames, a second probability is determined based on the second subset that the object includes the second part.

[0171] The multiple first probabilities and multiple second probabilities obtained by accumulation are respectively.

[0172] In one or more embodiments of this application, when the confidence module 1004 determines, based on a first subset and a second subset, that the object includes a first portion associated with the first subset and a second portion associated with the second subset, it is further configured to:

[0173] For each point cloud frame in the second set of point cloud frames, a first association probability is determined based on the first subset and the second subset, indicating that the object includes the first part.

[0174] For each point cloud frame in the second set of point cloud frames, a second association probability is determined based on the first subset and the second subset, indicating that the object includes the second part.

[0175] The multiple first association probabilities and multiple second association probabilities obtained by accumulation are respectively obtained.

[0176] In one or more embodiments of this application, when the confidence module 1004 determines, based on a first subset and a second subset, that the object includes a first portion associated with the first subset and a second portion associated with the second subset, it is further configured to:

[0177] Based on the pre-stored dataset, the accuracy is determined for the first probability and the second probability of each point cloud frame in the second set of point cloud frames.

[0178] In one or more embodiments of this application, the acquisition module 1001 is further configured to acquire a third set of point cloud frames associated with the surrounding environment collected by the sensing system.

[0179] The determining module 1003 is further configured to determine a set of points in each frame of the third set of point cloud frames that are associated with the second part of the object, as well as a third subset and a fourth subset of the set of points.

[0180] The confidence module 1004 is further configured to determine, based on the third subset and the fourth subset, the third confidence level of the second part comprising the third part associated with the third subset and the fourth confidence level of the fourth part associated with the fourth subset.

[0181] In one or more embodiments of this application, the modeling module 1002 is further configured to generate a representation of a third part of the object and a representation of a fourth part of the object in response to both the third confidence level and the fourth confidence level being higher than a predetermined threshold.

[0182] In one or more embodiments of this application, the object is a truck, the first part is the truck cab, and the second part is the truck trailer.

[0183] The apparatus for modeling objects in this application embodiment can be used to execute the technical solutions of the above method embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0184] Figure 11 A schematic diagram of the hardware structure of an electronic device according to an embodiment of this application is shown. Figure 11 As shown, the electronic device 1100 is used to implement the corresponding electronic device in any of the above method embodiments (e.g., Figure 1b The computing device 102 shown in this embodiment, the electronic device 1100 may include: a memory 1101, a processor 1102 and a communication interface 1103.

[0185] Memory 1101 is used to store computer programs. Memory 1101 may include high-speed random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0186] Processor 1102 is used to execute computer programs stored in memory to implement the methods in the above embodiments, as detailed in the relevant descriptions in the foregoing method embodiments. The processor 1102 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0187] Optionally, the memory 1101 can be either independent or integrated with the processor 1102. When the memory 1101 is a device independent of the processor 1102, the electronic device 1100 may also include a bus. This bus is used to connect the memory 1101 and the processor 1102. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0188] Communication interface 1103 is used to receive or send various data and / or instructions.

[0189] The electronic device provided in this embodiment can be used to execute the methods in the above embodiments. Its implementation and technical effects are similar, and will not be described again here.

[0190] In addition, this embodiment also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the methods described in the above embodiments.

[0191] In addition, this embodiment also provides a computer program product, including a computer program, which is executed by a processor to implement the methods described in the above embodiments.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0193] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0195] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0197] The above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them. Although the embodiments of this application have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0198] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0199] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for modeling objects, comprising: Acquire the first set of point cloud frames associated with the surrounding environment collected by the sensing system; Generate a hierarchical representation of objects in the surrounding environment based on the first set of point cloud frames. The hierarchical representation includes a first representation of the objects. Acquire a second set of point cloud frames associated with the surrounding environment, collected by the sensing system; Determine a set of points associated with the object in each frame of the second set of point cloud frames, as well as a first subset and a second subset of the set of points; The object is determined based on the first subset and the second subset to include a first confidence level of a first portion associated with the first subset and a second confidence level of a second portion associated with the second subset; Determine whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level; The second group of point cloud frames includes multiple point cloud frames. Determining the object based on the first subset and the second subset includes a first confidence level of a first portion associated with the first subset and a second confidence level of a second portion associated with the second subset, comprising: For each point cloud frame in the second set of point cloud frames, a first probability is determined based on the first subset that the object includes the first part. For each point cloud frame in the second set of point cloud frames, a second probability is determined based on the second subset that the object includes the second part. The first probability and the second probability are accumulated respectively to obtain the corresponding first confidence level and second confidence level.

2. The method according to claim 1, wherein, Determining whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level includes: In response to both the first confidence level and the second confidence level being lower than a predetermined threshold, the hierarchical representation remains unchanged.

3. The method according to claim 1, wherein, Determining the object based on the first subset and the second subset includes a first confidence level of a first portion associated with the first subset and a second confidence level of a second portion associated with the second subset, further comprising: For each point cloud frame in the second set of point cloud frames, a first association probability is determined based on the first subset and the second subset, indicating that the object includes the first part. For each point cloud frame in the second set of point cloud frames, a second association probability is determined based on the first subset and the second subset, indicating that the object includes the second part. The accumulated first association probabilities are multiplied by the accumulated first probabilities to obtain the first confidence level; and The accumulated second association probabilities are multiplied together to obtain the second confidence level.

4. The method according to claim 1, wherein, Determining the object based on the first subset and the second subset includes a first confidence level of a first portion associated with the first subset and a second confidence level of a second portion associated with the second subset, further comprising: Based on the pre-stored dataset, the accuracy is determined for the first probability and the second probability of each point cloud in the second set of point cloud frames. The first confidence level is obtained by multiplying the accumulated first probabilities by the accuracy corresponding to each first probability. The second confidence level is obtained by multiplying the accumulated second probabilities by the accuracy corresponding to each second probability.

5. The method according to claim 1, wherein, Determining whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level includes: In response to at least one of the first confidence level and the second confidence level being higher than a predetermined threshold, the hierarchical representation is changed.

6. The method according to claim 5, wherein, In response to at least one of the first confidence level and the second confidence level being higher than a predetermined threshold, changing the hierarchical representation includes: In response to one of the first confidence level and the second confidence level being lower than a predetermined threshold and the other being higher than a predetermined threshold, at least one of the size and shape of the first representation is changed.

7. The method according to claim 5, wherein, In response to at least one of the first confidence level and the second confidence level being higher than a predetermined threshold, changing the hierarchical representation includes: In response to both the first confidence level and the second confidence level being higher than a predetermined threshold, a second representation of the first part of the object and a third representation of the second part of the object are generated. The hierarchical representation is modified to further include a second representation of the first portion and a third representation of the second portion, wherein the second and third representations are at least partially located within the first representation.

8. The method according to claim 7, further comprising: Acquire a third set of point cloud frames associated with the surrounding environment, collected by the sensing system; Determine a set of points in each frame of the third set of point cloud frames that are associated with the second part of the object, as well as a third subset and a fourth subset of that set of points. The second part is determined based on the third subset and the fourth subset, comprising a third confidence level of the third part associated with the third subset and a fourth confidence level of the fourth part associated with the fourth subset; Based on the third and fourth confidence levels, determine whether to further modify the hierarchical representation of the object.

9. The method according to claim 8, wherein, Determining whether to further modify the hierarchical representation of the object based on the third confidence level and the fourth confidence level includes: In response to both the third and fourth confidence levels being higher than a predetermined threshold, a fourth representation of the third part of the object and a fifth representation of the fourth part of the object are generated. The hierarchical representation is modified to further include a fourth representation of the third part and a fifth representation of the fourth part, the fourth and fifth representations being at least partially located within the third representation.

10. The method according to claim 1, wherein, The object is a truck, the first part is the truck's tractor unit, and the second part is the truck's trailer.

11. A method for controlling a vehicle, comprising: Acquire the first set of point cloud frames associated with the surrounding environment collected by the sensing system; Generate a hierarchical representation of objects in the surrounding environment based on the first set of point cloud frames. The hierarchical representation includes a first representation of the objects. Acquire a second set of point cloud frames associated with the surrounding environment, collected by the sensing system; Determine a set of points associated with the object in each frame of the second set of point cloud frames, as well as a first subset and a second subset of the set of points; The object is determined based on the first subset and the second subset to include a first confidence level of a first portion associated with the first subset and a second confidence level of a second portion associated with the second subset; Determine whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level; Generate the path the vehicle will take based on the hierarchical representation of the object; Control the vehicle to travel along the generated path; The second group of point cloud frames includes multiple point cloud frames. Determining the object based on the first subset and the second subset includes a first confidence level of a first portion associated with the first subset and a second confidence level of a second portion associated with the second subset, comprising: For each point cloud frame in the second set of point cloud frames, a first probability is determined based on the first subset that the object includes the first part. For each point cloud frame in the second set of point cloud frames, a second probability is determined based on the second subset that the object includes the second part. The first probability and the second probability are accumulated respectively to obtain the corresponding first confidence level and second confidence level.

12. The method according to claim 11, wherein, Determining whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level includes: In response to both the first confidence level and the second confidence level being lower than a predetermined threshold, the hierarchical representation remains unchanged.

13. The method according to claim 11, wherein, Determining whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level includes: In response to at least one of the first confidence level and the second confidence level being higher than a predetermined threshold, the hierarchical representation is changed. The object-based hierarchical representation for generating the vehicle's path includes: The modified hierarchical representation based on objects generates the path that the vehicle will travel.

14. The method according to claim 13, wherein, In response to at least one of the first confidence level and the second confidence level being higher than a predetermined threshold, changing the hierarchical representation includes: In response to one of the first confidence level and the second confidence level being lower than a predetermined threshold and the other being higher than a predetermined threshold, at least one of the size and shape of the first representation is changed.

15. The method according to claim 13, wherein, In response to at least one of the first confidence level and the second confidence level being higher than a predetermined threshold, changing the hierarchical representation includes: In response to both the first confidence level and the second confidence level being higher than a predetermined threshold, a second representation of the first part of the object and a third representation of the second part of the object are generated. The hierarchical representation is modified to further include a second representation of the first portion and a third representation of the second portion, wherein the second and third representations are at least partially located within the first representation.

16. An apparatus for modeling an object, comprising: The acquisition module is used to acquire the first set of point cloud frames associated with the surrounding environment collected by the perception system; The modeling module is used to generate a hierarchical representation of objects in the surrounding environment based on the first set of point cloud frames, wherein the hierarchical representation includes a first representation of the objects; The acquisition module is also used to acquire a second set of point cloud frames associated with the surrounding environment collected by the perception system; The first determining module is used to determine a set of points associated with the object in each frame of the second set of point cloud frames, as well as a first subset and a second subset of the set of points. A confidence module is configured to determine, based on a first subset and a second subset, that the object includes a first portion associated with the first subset and a second portion associated with the second subset; The second determining module is used to determine whether to change the hierarchical representation of the object based on the first confidence level and the second confidence level; The second group of point cloud frames includes multiple point cloud frames. The confidence module is further used for: For each point cloud frame in the second set of point cloud frames, a first probability is determined based on the first subset that the object includes the first part. For each point cloud frame in the second set of point cloud frames, a second probability is determined based on the second subset that the object includes the second part. The first probability and the second probability are accumulated respectively to obtain the corresponding first confidence level and second confidence level.

17. An electronic device comprising: A processor and a memory, the memory storing program instructions; the processor executes the program instructions to implement the method as described in any one of claims 1-15.

18. A computer-readable storage medium storing program instructions that, when executed by a processor, are used to implement the method as claimed in any one of claims 1-15.

19. A computer program product comprising program instructions which, when executed by a processor, are configured to implement the method as claimed in any one of claims 1-15.

Images