Method and apparatus for processing a visual point cloud

Abstract

The embodiment of the present disclosure discloses a processing method and processing device of visual point cloud, wherein the processing method comprises: acquiring an image sequence collected by a vehicle-mounted camera in a preset time period; determining original point cloud data of the preset time period based on the image sequence; determining a weight of the original point cloud based on a time difference between an image acquisition time of the original point cloud and a target time; performing point cloud dense processing on the original point cloud data based on a position and the weight of the original point cloud, to obtain dense point cloud data; and determining an obstacle point cloud cost map within a preset range of the vehicle based on the dense point cloud data. The embodiment of the present disclosure can generate an obstacle point cloud cost map within a preset range of the vehicle, so as to select the most reasonable planning trajectory from at least one planning trajectory.

Description

Technical Field

[0001] This disclosure relates to the fields of assisted driving technology and computer vision technology, and in particular to a method and apparatus for processing visual point clouds. Background Technology

[0002] In assisted driving mode or autonomous driving mode, the vehicle can capture images of the surrounding environment through onboard cameras, determine information about obstacles near the vehicle based on the images captured by the onboard cameras, and then generate a planned trajectory for the vehicle.

[0003] Currently, when controlling vehicle movement based on a planned trajectory generated from point clouds identified from nearby images, collisions with obstacles sometimes occur. How to accurately obtain obstacle information from nearby images to generate a reasonable vehicle trajectory is a problem that urgently needs to be solved. Summary of the Invention

[0004] While the original point cloud obtained from the vehicle's environmental images captured by the vehicle-mounted camera has high accuracy, it is relatively sparse and difficult to accurately represent the edge positions of obstacles near the vehicle. Therefore, it is difficult to accurately determine the position and size of obstacles based on the original point cloud, and consequently, it is difficult to generate a reasonable vehicle planning trajectory.

[0005] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a method and apparatus for processing visual point clouds.

[0006] According to a first aspect of the present disclosure, a method for processing visual point clouds is provided, comprising:

[0007] Acquire image sequences captured by the vehicle-mounted camera within a preset time period;

[0008] Based on the image sequence, the original point cloud data for the preset time period is determined, wherein the original point cloud data includes the location of the original point cloud and the image acquisition time;

[0009] The weight of the original point cloud is determined based on the time difference between the image acquisition time and the target time, wherein the target time is the end acquisition time of the preset time period.

[0010] Based on the position and weight of the original point cloud, point cloud densification is performed on the original point cloud data to obtain dense point cloud data.

[0011] Based on the dense point cloud data, an obstacle point cloud cost map is determined at a distance of a preset range from the vehicle.

[0012] According to a second aspect of the present disclosure, a visual point cloud processing apparatus is provided, comprising:

[0013] The image sequence acquisition module is used to acquire image sequences captured by the vehicle-mounted camera within a preset time period;

[0014] The raw point cloud data determination module is used to determine the raw point cloud data of the preset time period based on the image sequence, wherein the raw point cloud data includes the location of the raw point cloud and the image acquisition time;

[0015] The point cloud weight determination module is used to determine the weight of the original point cloud based on the time difference between the image acquisition time and the target time, wherein the target time is the end acquisition time of the preset time period.

[0016] The dense point cloud data acquisition module is used to perform point cloud density processing on the original point cloud data based on the position and weight of the original point cloud to obtain dense point cloud data.

[0017] The obstacle point cloud cost map determination module is used to determine the obstacle point cloud cost map within a preset range near the vehicle based on the dense point cloud data.

[0018] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the visual point cloud processing method described in the first aspect above.

[0019] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising:

[0020] processor;

[0021] Memory used to store the processor's executable instructions;

[0022] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the visual point cloud processing method described in the first aspect above.

[0023] Based on the visual point cloud processing method and processing apparatus provided in the above embodiments of this disclosure, original point cloud data including obstacles near the vehicle within a preset time period is extracted from the image sequence acquired by the vehicle-mounted camera within a preset time period. The weight of the original point cloud is determined according to the time difference between the image acquisition time and the target time. Then, point cloud density processing is performed on the original point cloud data based on the position and weight of the original point cloud to obtain dense point cloud data that can accurately represent the position and size of obstacles near the vehicle. The obstacle point cloud cost map of the vehicle within a preset range determined by the dense point cloud data can accurately represent the obstacle cost incurred by the vehicle traveling along at least one planned trajectory between the initial position and the target position of the trajectory planning. This helps to select the most reasonable (i.e., the lowest cost) planned travel trajectory from at least one planned travel trajectory.

[0024] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0025] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0026] Figure 1 This is a flowchart illustrating a method for processing visual point clouds in one embodiment of this disclosure;

[0027] Figure 2 This is a flowchart illustrating step S4 in one embodiment of the present disclosure;

[0028] Figure 3 This is a schematic diagram illustrating the generation of a dense point cloud in one example of this disclosure;

[0029] Figure 4 This is a schematic diagram illustrating the generation of different planned trajectories from the original point cloud and the dense point cloud, respectively, in one example of this disclosure;

[0030] Figure 5 This is a flowchart illustrating step S5 in one embodiment of the present disclosure;

[0031] Figure 6 This is a schematic diagram of the vehicle driving position cost in one example of this disclosure;

[0032] Figure 7 This is a schematic diagram of the vehicle speed cost in one example of this disclosure;

[0033] Figure 8This is a schematic diagram of curvature cost in one example of this disclosure;

[0034] Figure 9 This is a schematic diagram of the relative length cost in one example of this disclosure;

[0035] Figure 10 This is a flowchart illustrating step S5-2 in one embodiment of this disclosure;

[0036] Figure 11 This is a flowchart illustrating the steps following step S5 in one embodiment of this disclosure.

[0037] Figure 12 This is a structural block diagram of a visual point cloud processing device in one embodiment of the present disclosure;

[0038] Figure 13 This is a structural block diagram of a dense point cloud data acquisition module 400 in one embodiment of this disclosure;

[0039] Figure 14 This is a structural block diagram of an obstacle point cloud cost map determination module 500 in one embodiment of this disclosure;

[0040] Figure 15 This is a structural block diagram of a visual point cloud processing apparatus in another embodiment of this disclosure;

[0041] Figure 16 This is a structural block diagram of an electronic device provided in one embodiment of the present disclosure. Detailed Implementation

[0042] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0043] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0044] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0045] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0046] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0047] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0048] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0049] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0050] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0051] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0052] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0053] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0054] Application Overview

[0055] In developing this disclosure, the inventors discovered that by capturing images of the environment near a vehicle using an onboard camera, raw point cloud data of the surrounding environment can be obtained. The raw point cloud data is a feature point cloud extracted from the image and includes obstacles near the vehicle. Combined with camera parameters, the location of the feature point cloud can be accurately determined. However, because the feature point cloud is relatively sparse, it cannot cover the entire location of obstacles, especially not all their edges. Therefore, the raw point cloud cannot accurately represent the edge positions of obstacles near the vehicle, making it difficult to accurately determine the location and size of obstacles based on it. If the raw point cloud is clustered, the resulting obstacle detection boxes, while able to reconstruct the length and width of obstacles using some sparse data, suffer from inaccurate detection boxes, leading to poor final accuracy and hindering reasonable vehicle trajectory planning.

[0056] In the embodiments of this disclosure, images of the environment near the vehicle are captured by an onboard camera. Based on the captured images, raw point cloud data of the environment near the vehicle can be obtained. The weight of the raw point cloud is determined based on the time difference between the image acquisition time and the target time. Then, based on the position and weight of the raw point cloud, the raw point cloud data is processed to obtain dense point cloud data that can accurately represent the position and size of obstacles near the vehicle. The obstacle point cloud cost map of the vehicle within a preset range determined by the dense point cloud data can accurately represent the obstacle cost incurred by the vehicle when traveling along at least one planned trajectory between the initial position and the target position of the trajectory planning. This helps to select the most reasonable planned driving trajectory from at least one planned driving trajectory.

[0057] Exemplary methods

[0058] Figure 1 This is a flowchart illustrating a method for processing visual point clouds in one embodiment of this disclosure. Figure 1 As shown, it may include the following steps:

[0059] S1: Acquire the image sequence captured by the vehicle-mounted camera within a preset time period.

[0060] The vehicle can capture images of the environment within a preset range near the vehicle within a preset time period, whether the vehicle is stationary or in motion (at a speed less than a preset low-speed threshold).

[0061] The vehicle-mounted camera can be a panoramic camera or multiple monocular cameras installed on the vehicle to collect images of the surrounding environment. The image sequence can be obtained by the panoramic camera or by the simultaneous capture of images by multiple monocular cameras and the image sequence can be obtained by image fusion.

[0062] The preset time period can be a time period N seconds prior to the current time. Where N is an integer greater than 1.

[0063] The preset range near the vehicle can be a cylindrical area with radius R and height H centered on the vehicle's location.

[0064] In one example of this disclosure, when there are obstacles such as other vehicles, walls, or roadblocks between the vehicle's current position and the target position, the driver of the vehicle may find it difficult to maneuver the vehicle from the current position to the target position. In this case, the occupants of the vehicle can use an onboard camera to capture a sequence of images N seconds prior to the current time, while the vehicle is parked or moving at low speed.

[0065] S2: Based on the image sequence, determine the raw point cloud data for a preset time period. The raw point cloud data includes the location of the raw point cloud and the time of image acquisition.

[0066] M frames can be selected from an image sequence, and then visual point cloud extraction can be performed on the M frames to obtain raw point cloud data including the location of the original point cloud and the time of image acquisition. The M frames can include all image frames or some keyframes of the image sequence.

[0067] S3: Determine the weight of the original point cloud based on the time difference between the image acquisition time and the target time. The target time is the end acquisition time of the preset time period.

[0068] A pre-defined relationship between time difference and weights is established. The longer the time between the image acquisition time of the original point cloud and the target time, the greater the deviation of the original point cloud's position at the image acquisition time relative to its position at the target time. Therefore, the weights of the original point cloud can be set to change with the time difference. For example, if the target time is t2, the image acquisition time of a point cloud in the original point cloud is t1, and the time difference between the image acquisition time t1 and the target time t2 is x, then the relationship between the weight y of that point cloud and the time difference x can be determined by the following formula: y = f(x), where f can be expressed as a function with the time difference x as the independent variable and the weight y as the dependent variable. For example, when y = k1*x, the function represented by f is a proportional relationship between the time difference x and the weight y with k as the proportion, where k1 is a pre-defined first proportionality coefficient.

[0069] After calculating the time difference between the image acquisition time and the target time for all points in the original point cloud, a matching is performed in the correspondence between time difference and weight based on the calculated time difference. The weight of all points in the original point cloud can be determined based on the matching result.

[0070] S4: Perform point cloud densification processing on the original point cloud data based on the position and weight of the original point cloud to obtain dense point cloud data.

[0071] The original point cloud includes point clouds of key points of obstacles near the vehicle extracted from the visual point cloud, and may also include point clouds of other objects near the vehicle (i.e., point clouds of objects that do not affect vehicle trajectory planning, such as point clouds of objects outside the vehicle trajectory planning range, or point clouds of suspended signs within the vehicle trajectory planning spatial range but whose lower end is higher than the vehicle height). The original point cloud has high positional accuracy, but it is relatively sparse and does not include point clouds of all edge positions of obstacles near the vehicle.

[0072] To reduce the probability of vehicles colliding with the edges of obstacles while traveling along the planned trajectory, point cloud densification can be performed based on the position and weight of at least one point cloud in the original point cloud. That is, the range of the original point cloud is expanded accordingly based on the position and weight of at least one point cloud in the original point cloud to obtain dense point cloud data that includes the edge positions of obstacles.

[0073] S5: Based on dense point cloud data, determine the obstacle point cloud cost map within a preset range of the vehicle.

[0074] Since the dense point cloud data can include the point cloud data of obstacles in the range near the vehicle, an obstacle point cloud cost map of a preset range of the vehicle can be generated according to the positions and weights of the dense point clouds in the dense point cloud data. Among them, the weights of the dense point clouds include the weights of the original point clouds and the weights of the increased positions of the dense point clouds relative to the original point clouds. The obstacle point cloud cost map can include the cost information of the vehicle passing through the position where the dense point cloud is located. Among them, the cost information in the obstacle point cloud cost map is the obstacle cost paid by the vehicle for passing through the position in the obstacle point cloud cost map.

[0075] In an example of the present disclosure, when a certain trajectory point in the planned trajectory passes through the position where an obstacle is located, it can be determined that the obstacle cost of this trajectory point is infinite, that is, when the vehicle is controlled to travel according to the planned trajectory, the vehicle will collide with the obstacle at this trajectory point.

[0076] When the distance between a certain trajectory point in the planned trajectory and the position where an obstacle is located is less than the preset safety distance, when the vehicle is controlled to travel according to the planned trajectory, the vehicle may collide with the obstacle at this trajectory point. It is possible to set the obstacle cost of this trajectory point according to the distance relationship between this trajectory point and the distance between the nearby obstacles and the preset safety distance. Among them, the obstacle cost can be understood as the cost that the vehicle needs to pay for possible collisions or scratches with nearby obstacles when passing through this trajectory point. The larger the cost value, the greater the probability of vehicle accidents such as collisions or scratches when the vehicle travels through this trajectory point.

[0077] In an example of the present disclosure, the obstacle cost of the trajectory points in the planned trajectory can be determined by the following method:

[0078] When S≥S1, Z = Z1;

[0079] When S≤S2, Z = Z2;

[0080] When S2 < S < S1, Z = k2*(S1 - S)*(Z2 - Z1).

[0081] Among them, S is the distance between a trajectory point in the planned trajectory and the nearest obstacle nearby, S1 is the preset safety distance, S2 is the preset collision distance, Z is the obstacle cost of this trajectory point, Z1 is the preset minimum value of the obstacle cost, Z2 is the preset maximum value of the obstacle cost, and k2 is the preset second proportionality coefficient.

[0082] When the obstacle cost Z = Z1 at a certain trajectory point, it means that the vehicle will not collide or scrape with nearby obstacles when it reaches that trajectory point. When the obstacle cost Z = Z2 at a certain trajectory point, it means that the vehicle will inevitably collide or scrape with nearby obstacles when it reaches that trajectory point. When the obstacle cost at a certain trajectory point is a value between Z1 and Z2 (excluding Z1 and Z2), it means that the vehicle may collide or scrape with nearby obstacles when it reaches that trajectory point. The higher the value of the obstacle cost Z, the higher the probability of a collision or scrape.

[0083] In a specific example of this disclosure, S1 is 1 meter, S2 is 0 meters, Z1 is 0, Z2 is 100, and k2 is 100. When the distance S between a trajectory point and the nearest obstacle is 1.2 meters, the obstacle cost Z = Z1 = 0; when the distance S between a trajectory point and the nearest obstacle is 0.8 meters, the obstacle cost Z = 100 * (1 - 0.8) * (100 - 0) = 20; when the distance S between a trajectory point and the nearest obstacle is 0 meters, the obstacle cost Z = Z2 = 100.

[0084] When the distance between a certain trajectory point in the planned trajectory and the location of nearby obstacles is greater than the preset safety distance, the vehicle will not collide with nearby obstacles when controlling the vehicle to drive according to the trajectory. At this time, the obstacle cost of that trajectory point can be set to 0.

[0085] In this embodiment, images of the environment near the vehicle are captured by an onboard camera. Based on the captured images, raw point cloud data of the environment near the vehicle can be obtained. The weight of the raw point cloud is determined based on the time difference between the image acquisition time and the target time. Then, based on the position and weight of the raw point cloud, the raw point cloud data is processed to obtain dense point cloud data that can accurately represent the position and size of obstacles near the vehicle. An obstacle point cloud cost map within a preset range of the vehicle is determined based on the dense point cloud data.

[0086] In this embodiment, raw point cloud data including obstacles near the vehicle within a preset time period is extracted from the image sequence captured by the vehicle-mounted camera within a preset time period. The weight of the raw point cloud is determined based on the time difference between the image acquisition time and the target time. Then, point cloud density processing is performed on the raw point cloud data based on the position and weight of the raw point cloud to obtain dense point cloud data that can accurately represent the position and size of obstacles near the vehicle. The obstacle point cloud cost map within a preset range of the vehicle determined based on the dense point cloud data can accurately represent the cost incurred by the vehicle to travel along at least one planned trajectory between the initial position and the target position of the trajectory planning, which helps to select the most reasonable planned driving trajectory from at least one planned driving trajectory.

[0087] In one embodiment of this disclosure, the preset time period is a time period of preset length preceding the current time as the end time of data collection, and the weight of the original point cloud is inversely correlated with the time difference.

[0088] Since the longer the time between the image acquisition time of the original point cloud and the current time is, the less accurate the representation of the relative positional relationship between the obstacle position in the original point cloud and the current position of the vehicle is, the weight of the original point cloud is set to be inversely correlated with the time difference.

[0089] In one example of this disclosure, image frames A, B, and C in the image sequence all contain obstacles M. The time difference between image frame A and the current time is N1 seconds, the time difference between image frame B and the current time is N2 seconds, and the time difference between image frame C and the current time is N3 seconds. The weight of the original point cloud in image frame A is y1, the weight of the original point cloud in image frame B is y2, and the weight of the original point cloud in image frame C is y3. N1, N2, and N3 are all integers greater than 0. When N1 > N2 > N3, y1 < y2 < y3 can be set.

[0090] In this embodiment, the greater the time difference between the image acquisition time of the original point cloud and the current time, the greater the probability of changes in the relative positional relationship between the vehicle and the obstacle (for example, the vehicle's position changes or the position of movable obstacles (such as other vehicles or pedestrians near the vehicle) changes, resulting in changes in the relative positional relationship between the vehicle and the obstacle). This leads to lower accuracy in the relative positional relationship between the obstacle position in the original point cloud and the vehicle's current position. Therefore, the weight of the original point cloud is set to be inversely correlated with the time difference between the image acquisition time of the original point cloud and the target time, so that subsequent steps can obtain a high-precision obstacle point cloud cost map.

[0091] In one embodiment of this disclosure, the weight of the original point cloud is either linearly inversely correlated with the time difference between the image acquisition time and the target time, or inversely correlated with the time difference stage.

[0092] In one example of this disclosure, the weight of any point in the original point cloud can be determined using the following formula:

[0093] x = t2 - t1

[0094] y = k1*t2 - t1

[0095] Where t1 is the image acquisition time of a point cloud in the original point cloud, t2 is the current time, x is the time difference between the image acquisition time of the point cloud and the current time, y is the weight of the point cloud, and k1 is the preset first proportional coefficient.

[0096] In another example of this disclosure, the weight of any point in the original point cloud can also be determined using the following piecewise function:

[0097] x = t2 - t1

[0098]

[0099] Where t1 is the image acquisition time of a point cloud in the original point cloud, t2 is the current time, x is the time difference between the image acquisition time of that point cloud and the current time, y is the weight of that point cloud, and t A t B t C and t D These represent the pre-set first time difference threshold, second time difference threshold, third time difference threshold, and fourth time difference threshold, respectively, and t A <t B <t C <t D y1, y2, y3 and y4 are the first weight value, the second weight value, the third weight value and the fourth weight value, respectively, and y1 > y2 > y3 > y4.

[0100] In a specific example of this disclosure, t is set A For 0 seconds, t B For 0.5 seconds, t C For 1 second, t DThe time interval is set to 2 seconds, with y1 set to 1, y2 to 0.75, y3 to 0.5, and y4 to 0.25. The weight of the original point cloud of the image at the current moment is 1; the weight of the original point cloud of the image within 0.5 seconds of the current moment is 0.75, the weight of the original point cloud of the image between 0.5 and 1 second from the current moment is 0.5, and the weight of the image between 1 and 2 seconds from the current moment is 0.25. Furthermore, the weight of the original point cloud of the image more than 2 seconds from the current moment can be set to 0.

[0101] In this embodiment, when the current time is the termination time of the preset time period, the weight of the original point cloud and the time difference between the image acquisition time of the original point cloud and the target time show one of linear inverse correlation and inverse correlation in different time difference stages. This can effectively characterize the accuracy of the relative position between the obstacle in the original point cloud and the current position of the vehicle, so that subsequent steps can obtain a high-precision obstacle point cloud cost map.

[0102] Figure 2 This is a flowchart illustrating step S4 in one embodiment of this disclosure. Figure 2 As shown, step S4 may include:

[0103] S4-1: Determine the pixel values ​​of the original point cloud based on the weights of the original point cloud.

[0104] The correspondence between point cloud weights and pixel values ​​is predefined. For example, when the weight of the original point cloud is the first weight value, the corresponding pixel value is the first pixel value; when the weight of the original point cloud is the second weight value, the corresponding pixel value is the second pixel value; when the weight of the original point cloud is the third weight value, the corresponding pixel value is the third pixel value, and so on. For instance, when the first weight value is 1, the RGB value of the first pixel value can be (0, 0, 0); when the second weight value is 0.75, the RGB value of the second pixel value can be (63, 63, 63); when the third weight value is 0.5, the RGB value of the third pixel value can be (127, 127, 127); and when the fourth weight value is 0.25, the RGB value of the fourth pixel value can be (191, 191, 191). Those skilled in the art will understand that the specific values ​​of the weights and pixel values ​​are not fixed and can be adjusted accordingly.

[0105] After obtaining the weights of the original point cloud, the pixel values ​​of the original point cloud can be determined by matching the weights of the original point cloud with the corresponding relationship between the weights and pixel values.

[0106] Figure 3 This is a schematic diagram illustrating the generation of a dense point cloud in one example of this disclosure. For example... Figure 3As shown, when the weight of the original point cloud is y, and it exhibits an inverse correlation with the time difference x between the image acquisition time and the target time, the preset relationship between the weight y and the pixel value C is as follows:

[0107] When y = y1, the pixel value of the original point cloud = C1 (i.e., the first pixel value mentioned above);

[0108] When y = y2, the pixel value of the original point cloud = C2 (i.e., the second pixel value mentioned above);

[0109] When y = y3, the pixel value of the original point cloud = C3 (i.e., the third pixel value mentioned above);

[0110] When y = y4, the pixel value of the original point cloud is C4 (i.e., the fourth pixel value mentioned above).

[0111] S4-2: Based on the position and pixel values ​​of the original point cloud, perform Gaussian blur processing on the original point cloud to obtain blurred point cloud data. The Gaussian blur processing includes point cloud range adjustment and point cloud pixel value adjustment. The blur range and color can be changed by controlling the size and standard deviation of the Gaussian kernel. The blurred point cloud data includes the position and pixel values ​​of the blurred point cloud.

[0112] Continue to refer to Figure 3 The system uses a preset Gaussian blur method to adjust the point cloud range and pixel values ​​based on the position and pixel values ​​of the original point cloud, thus obtaining blurred point cloud data.

[0113] S4-3: Based on the position and pixel values ​​of the original point cloud and the position and pixel values ​​of the blurred point cloud, perform point cloud overlay processing on the original point cloud and blurred point cloud data to obtain dense point cloud data.

[0114] Continue to refer to Figure 3 In dense point cloud data, for pixel values ​​at locations where the original point cloud and the fuzzy point cloud overlap, the pixel values ​​of the original point cloud can be used as the pixel values ​​of the dense point cloud at the overlapping locations; for pixel values ​​at locations where the original point cloud and the fuzzy point cloud do not overlap, the pixel values ​​of the corresponding point cloud (e.g., the original point cloud or the fuzzy point cloud) at the non-overlapping locations can be used.

[0115] Figure 4 This is a schematic diagram illustrating the generation of different planned trajectories from the original point cloud and the dense point cloud, respectively, in one example of this disclosure. For example... Figure 4 As shown, compared to generating a trajectory based on the original point cloud, the corresponding trajectory points in the trajectory generated based on the dense point cloud (i.e., original + fuzzy point cloud) are farther away from nearby obstacles, which can reduce the probability of vehicle collision with obstacles, making it safer for vehicles to drive according to the planned trajectory generated by the dense point cloud, and the trajectory planning is more reasonable.

[0116] In this embodiment, by setting the parameters of Gaussian blur processing, the dense point cloud data obtained by Gaussian blur processing and point cloud overlay processing based on the position and weight value of the original point cloud can be expanded accordingly, and the pixel value of the original point cloud can be adjusted so that a reasonable obstacle point cloud cost map can be obtained in subsequent steps.

[0117] Figure 5 This is a flowchart illustrating step S5 in one embodiment of this disclosure. Figure 5 As shown, step S5 may include:

[0118] S5-1: Determine the weight of obstacle cost in the cost of trajectory planning for vehicles.

[0119] In addition to obstacle costs, trajectory planning costs can also include at least one of the following: vehicle position cost, vehicle speed cost, curvature cost, and relative length cost.

[0120] Vehicle position cost represents the cost incurred by a vehicle in achieving its planned position within the planned driving lane. This embodiment can be configured such that the vehicle position cost is positively correlated with the lateral distance of a trajectory point from the lane centerline. The smaller the lateral distance of a trajectory point from the lane centerline, the smaller the vehicle position cost; conversely, the larger the lateral distance, the greater the vehicle position cost.

[0121] In one example of this disclosure, the cost of a vehicle's travel position can be determined in the following way:

[0122] When the trajectory point of the planned trajectory is located on the center line of the lane, the vehicle driving position cost PA is set to the minimum value PA1 of the vehicle driving position cost;

[0123] When the trajectory point of the planned trajectory is located on the solid line between two adjacent lanes, the vehicle driving position cost PA is set to the maximum value PA2 of the vehicle driving position cost.

[0124] When the trajectory point of the planned trajectory is located on the dashed line between two adjacent lanes, the vehicle driving position cost PA is set to the median value PA3 of the vehicle driving position cost.

[0125] When the trajectory point of the planned trajectory is located between the lane center line and the nearest lateral solid line, the vehicle driving position cost PA is set to k3*(L2 / L1)*(PA2-PA1), where k3 is a pre-set third proportional coefficient, L1 is the lateral distance between the lane center line and the nearest lateral solid line, and L2 is the lateral distance between the vehicle and the lane center line.

[0126] When the trajectory points of the planned trajectory are located between the center line of the lane and the nearest lateral dotted line, set the vehicle driving position cost PA = k3 * (L3 / L1) * (PA3 - PA1), where L3 is the lateral distance between the vehicle and the nearest lateral dotted line.

[0127] Figure 6 It is a schematic diagram of the vehicle driving position cost in an example of the present disclosure. As Figure 6 shown, the minimum value PA1 can be set to 0, the maximum value PA2 to 100, and the intermediate value PA3 to 50. In the order from left to right, set the four planned trajectories as the first planned trajectory, the second planned trajectory, the third planned trajectory, and the fourth planned trajectory respectively.

[0128] Among them, both trajectory points on the first planned trajectory are located between the center line of the left lane and the left solid line, and the vehicle driving position cost PA of these two trajectory points can be calculated according to the formula PA = k3 * (L2 / L1) * (PA2 - PA1).

[0129] Both trajectory points on the second planned trajectory are located between the center line of the left lane and the intermediate dotted line, and the vehicle driving position cost PA of these two trajectory points can be calculated according to the formula PA = k3 * (L3 / L1) * (PA3 - PA1).

[0130] Both trajectory points on the third planned trajectory are located between the center line of the right lane and the intermediate dotted line, and the vehicle driving position cost PA of these two trajectory points can be calculated according to the formula PA = k3 * (L3 / L1) * (PA3 - PA1).

[0131] Both trajectory points on the fourth planned trajectory are located between the center line of the right lane and the right solid line, and the vehicle driving position cost PA of these two trajectory points can be calculated according to the formula PA = k3 * (L2 / L1) * (PA2 - PA1).

[0132] The vehicle speed cost represents the cost required for the planned vehicle speed relative to the speed limit value of the lane where the planned trajectory is located. In this embodiment, it can be set that: on the premise of not exceeding the speed limit value of the current driving lane of the vehicle, the vehicle speed cost has a positive relationship with the vehicle speed difference between the vehicle speed and the speed limit value of the current driving lane.

[0133] In an example of the present disclosure, the vehicle speed cost can be determined by the following method:

[0134] When V > k4 * V1, PB = PB1;

[0135] When V < k5 * V1, PB = PB2;

[0136] When k5 * V1 ≤ V ≤ k4 * V1, PB = PB2 * |V - V1|.

[0137] Where V is the vehicle speed at a certain trajectory point in the planned trajectory, V1 is the maximum speed limit of the lane where the trajectory point is located, k4 is the pre-set fourth proportional coefficient, k5 is the pre-set fifth proportional coefficient, PB is the speed cost, PB1 is the maximum speed cost, and PB2 is the reference value of the speed cost.

[0138] Figure 7 This is a schematic diagram illustrating the vehicle speed cost in one example of this disclosure. For example... Figure 7 As shown, V1 can be set to 80 km / h, k4 to 1.1, k5 to 0.5, and PB1 to 10. 7 PB2 is 100. When the vehicle speed V at a certain trajectory point in the planned trajectory is less than 40 km / h (0.5 * 80 km / h), it can be determined that the vehicle speed is too low and obstructs traffic. The speed cost PB at this trajectory point is 100 (i.e., PB2), which is the maximum speed cost within the reasonable speed range. When the vehicle speed V at a certain trajectory point in the planned trajectory is greater than 88 km / h, it can be determined that the vehicle speed is too high and is likely to cause a traffic accident. The speed cost at this trajectory point is 10. 7 (i.e., PB1, representing an unreasonable vehicle speed with a huge cost); when the vehicle speed V of a certain trajectory point in the planned trajectory is between 40 km / h and 88 km / h, the vehicle speed of the trajectory point can be calculated by PB = PB2 * |V - V1|.

[0139] Curvature cost represents the cost required to achieve the curvature of the planned trajectory. Due to considerations of driving comfort, drivers typically prioritize straight-line driving unless there are special circumstances (e.g., no obstacles in front of the vehicle). Therefore, this embodiment can be configured such that curvature and curvature cost are positively correlated: the greater the curvature of a trajectory point, the greater the curvature cost; conversely, the smaller the curvature of a trajectory point, the smaller the curvature cost.

[0140] In one example of this disclosure, the curvature cost can be determined as follows: PC = PC1 * (θ / θ1). Where PC is the curvature cost, PC1 is the curvature cost reference value, θ is the curvature of the trajectory line between a certain trajectory point and the previous trajectory point in the planned trajectory, and θ1 is the preset maximum curvature.

[0141] Figure 8 This is a schematic diagram of curvature cost in one example of this disclosure. For example... Figure 8As shown, the grid points represent the vehicle's initial position, and the data within the trajectory points represents the curvature cost. Three planned trajectories are defined from left to right: the first planned trajectory, the second planned trajectory, and the third planned trajectory. The trajectory points for each planned trajectory are defined as the first trajectory point, the second trajectory point, the third trajectory point, and the fourth trajectory point, respectively, following the direction from the initial position to the direction away from the initial position. The curvature cost reference value PC1 can be set to 100. The curvature of the trajectory lines between adjacent trajectory points can be obtained by curvature detection of the trajectory lines. The maximum curvature θ1 can be determined based on the vehicle's minimum turning radius; for example, θ1 can be set to 1 / R, where R is the minimum turning radius.

[0142] In the first planned trajectory, the curvature of the trajectory line between the first trajectory point and the initial trajectory point, and the curvature of the trajectory line between the second trajectory point and the first trajectory point, are both non-zero; the curvature of the trajectory line between the third trajectory point and the second estimated point, and the curvature of the trajectory line between the fourth trajectory point and the third trajectory point, are both zero. The curvature cost of each trajectory point can be calculated using PC = PC1 * (θ / θ1).

[0143] In the second planned trajectory, the curvature of the trajectory lines between adjacent trajectory points is 0, so the curvature cost of all trajectory points can be set to 0.

[0144] The curvature cost of trajectory points in the third planning trajectory is determined in a similar way to the curvature cost of trajectory points in the first planning trajectory, and will not be elaborated further.

[0145] Relative length cost represents the cost required for a vehicle to travel from the target sampling point to the farthest sampling point within the planned trajectory, relative to the trajectory length to the farthest sampling point. The farthest sampling point is the sampling point furthest from the vehicle's initial planned position within the planned trajectory. This embodiment can be configured such that the path length from the vehicle to the target sampling point is inversely related to the path length from the target sampling point to the farthest sampling point within the planned trajectory. The closer the target sampling point is to the farthest sampling point of the corresponding planned trajectory, the lower the relative length cost; conversely, the farther the target sampling point is from the farthest sampling point of the corresponding planned trajectory, the higher the relative length cost.

[0146] In one example of this disclosure, the relative length cost can be determined as follows: PD = PD1 * (d1 - d) / d1. Where PD is the relative length cost of a trajectory point in the planned trajectory, PC1 is the reference value of the relative length cost, d is the distance between the trajectory point in the planned trajectory and the initial position, and d1 is the length of the planned trajectory.

[0147] Figure 9 This is a schematic diagram illustrating the relative length cost in one example of this disclosure. For example... Figure 9As shown, the grid points represent the initial positions of the vehicles, and the relative length cost reference value PC1 can be set to 100. The relative length cost of the trajectory points (S3L0 / S3L1 / S3L2) at the sampling ends of the three planned trajectories is 0, which encourages vehicles to travel as far as possible; the relative length cost of the initial positions of the three planned trajectories is 100, which discourages vehicles from staying in place; the relative length cost of other trajectory points in the planned trajectories, such as the relative length cost of trajectory point (S1L0 / S1L1 / S1L2) is 66, and the relative length cost of trajectory point (S2L0 / S2L1 / S2L2) is 33.

[0148] The purpose of setting a relative length cost is to increase the ability to control lane-changing behavior during the evaluation process. Without this cost, vehicles would be unable to change lanes when faced with stationary obstacles.

[0149] The weighting of obstacle costs can be set to a predetermined value. For example, when the cost of trajectory planning includes obstacle cost, map cost, vehicle speed cost, curvature cost, and relative length cost, the weighting of obstacles can be set to 0.2. In this case, the total weighting of map cost, vehicle speed cost, curvature cost, and relative length cost is 0.8.

[0150] S5-2: Determine the obstacle point cloud cost map based on the weighted proportion of obstacle costs, the weight of the original point cloud, and its location. The obstacle point cloud cost map includes the location of the obstacle point cloud and the cost incurred by the vehicle traveling to that location.

[0151] The cost of the original point cloud can be calculated based on the weight of the obstacle cost and the weight of the original point cloud. Then, based on the cost of the original point cloud and its weight, the point cloud cost in the obstacle point cloud cost map can be determined.

[0152] In this embodiment, when planning the trajectory of a vehicle, in addition to obstacles within the trajectory planning area, the vehicle's position within the planned driving lane, the relationship between the vehicle speed and the lane speed limit, the curvature of the planned trajectory, and the path length from the vehicle to the target sampling point relative to the trajectory length to the farthest sampling point all affect the trajectory planning output. Therefore, based on the weight ratio of obstacle cost, the weight of the original point cloud, and its position, the obstacle point cloud cost map can be reasonably determined.

[0153] Figure 10 This is a flowchart illustrating step S5-2 in one embodiment of this disclosure. Figure 10 As shown, step S5-2 may include:

[0154] S5-2-1: Determine the weight of the obstacle point cloud based on the weight of the original point cloud and the weight ratio of the obstacle cost.

[0155] When planning vehicle trajectories, in addition to obstacle costs, map costs, speed costs, curvature costs, and relative length costs can also be considered. The weighting of obstacle costs characterizes the degree to which obstacles affect vehicle trajectory planning. Furthermore, the weighting of map costs characterizes the degree to which the vehicle's position within the planned lane affects trajectory planning. The weighting of speed costs characterizes the degree to which vehicle speed affects trajectory planning. The weighting of curvature costs characterizes the degree to which the curvature of the planned trajectory affects trajectory planning. The weighting of relative length costs characterizes the degree to which path length affects trajectory planning.

[0156] Obstacle point clouds are extracted from the original point cloud, and then the weights of the obstacle point clouds with respect to the time difference (between the image acquisition time of the original point cloud and the target time) are obtained. Then, the weights of the obstacle point clouds with respect to the time difference and the weights of the obstacle costs are multiplied to obtain the final weights of the obstacle point clouds in vehicle trajectory planning.

[0157] S5-2-2: Determine the location of the obstacle point cloud based on the location of the original point cloud. Since the location of the original point cloud is known, the location of the obstacle point cloud extracted from the original point cloud can also be determined.

[0158] S5-2-3: Determine the obstacle point cloud cost map based on the weights and positions of the obstacle point clouds. That is, the obstacle point cloud cost map is composed of the positions and weights of all obstacle point clouds.

[0159] In this embodiment, the obstacle point cloud cost map can be accurately determined based on the position and weight of the original point cloud, combined with the weight ratio of obstacle cost.

[0160] Figure 11 This is a flowchart illustrating the steps following step S5 in one embodiment of this disclosure. Figure 11 As shown, after step S5, the following may also be included:

[0161] S6: Based on the vehicle's initial and target positions, extract obstacle point cloud cost information for the target region, including both the initial and target positions, from the obstacle point cloud cost map. The target region can be a rectangular region. For example, the line segment between the initial and target positions can be used as the diagonal of the rectangular region. Alternatively, the line segment between the initial and target positions can be extended along its direction, and the extended line segment can be used as the diagonal of the rectangular region.

[0162] S7: Based on the obstacle point cloud cost information of the initial position, target position, and target area, the vehicle's driving trajectory is planned to determine the planned driving trajectory from the initial position to the target position. In the trajectory planning process, in addition to referring to obstacle costs, at least one of the following can also be referenced: map cost, vehicle speed cost, curvature cost, and relative length cost.

[0163] When there is only one planned driving trajectory, this planned driving trajectory is taken as the final planned driving trajectory; when there are multiple planned driving trajectories, the planned driving trajectory with the lowest obstacle cost can be taken as the final planned driving trajectory, or the planned driving trajectory with the lowest total cost of trajectory planning can be taken as the final planned driving trajectory.

[0164] In this embodiment, the planned driving trajectory of the vehicle from the initial position to the target position can be reasonably determined based on the obstacle point cloud cost of the target area, including the initial position and the target position.

[0165] Any of the visual point cloud processing methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the visual point cloud processing methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the visual point cloud processing methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0166] Exemplary device

[0167] Figure 12 This is a structural block diagram of a visual point cloud processing apparatus according to one embodiment of this disclosure. Figure 12 As shown, the visual point cloud processing device may include:

[0168] The image sequence acquisition module 100 is used to acquire image sequences captured by the vehicle-mounted camera within a preset time period;

[0169] The raw point cloud data determination module 200 is used to determine the raw point cloud data of the preset time period based on the image sequence, wherein the raw point cloud data includes the location of the raw point cloud and the image acquisition time.

[0170] The point cloud weight determination module 300 is used to determine the weight of the original point cloud based on the time difference between the image acquisition time and the target time, wherein the target time is the end acquisition time of the preset time period.

[0171] The dense point cloud data acquisition module 400 is used to perform point cloud density processing on the original point cloud data based on the position and weight of the original point cloud to obtain dense point cloud data.

[0172] The obstacle point cloud cost map determination module 500 is used to determine the obstacle point cloud cost map within a preset range near the vehicle based on the dense point cloud data.

[0173] In one embodiment of this disclosure, the preset time period is a time period extending from the current time as the end time of data collection by a preset time length, and the weight of the original point cloud is inversely correlated with the time difference.

[0174] In one embodiment of this disclosure, the weights of the original point cloud are either linearly inversely correlated with the time difference or inversely correlated across different time difference stages.

[0175] Figure 13 This is a structural block diagram of a dense point cloud data acquisition module 400 in one embodiment of this disclosure. For example... Figure 13 As shown, the dense point cloud data acquisition module 400 may include:

[0176] The pixel value determination unit 410 is used to determine the pixel value of the original point cloud based on the weight of the original point cloud.

[0177] The blur processing unit 420 is used to perform Gaussian blur processing on the original point cloud based on the position and pixel value of the original point cloud to obtain blurred point cloud data. The Gaussian blur processing includes expanding the position range of the point cloud and adjusting the pixel value of the point cloud. The blurred point cloud data includes the position and pixel value of the blurred point cloud.

[0178] The dense point cloud determination unit 430 is used to perform point cloud overlay processing on the original point cloud and the fuzzy point cloud data based on the position and pixel value of the original point cloud and the position and pixel value of the fuzzy point cloud to obtain the dense point cloud data.

[0179] Figure 14 This is a structural block diagram of an obstacle point cloud cost map determination module 500 in one embodiment of this disclosure. For example... Figure 14 As shown, the obstacle point cloud cost map determination module 500 may include:

[0180] The weighting ratio determination unit 510 is used to determine the weighting ratio of obstacle cost in the cost of trajectory planning for the vehicle.

[0181] The obstacle point cloud cost map determination unit 520 is used to determine the obstacle point cloud cost map based on the weight ratio of the obstacle cost, the weight and position of the original point cloud, wherein the obstacle point cloud cost map includes the obstacle point cloud position and the cost of the vehicle traveling to the obstacle point cloud position.

[0182] In one embodiment of this disclosure, the obstacle point cloud cost map determination unit 520 is used to determine the weight of the obstacle point cloud based on the weight of the original point cloud and the weight ratio of the obstacle cost; the obstacle point cloud cost map determination unit 520 is also used to determine the position of the obstacle point cloud based on the position of the original point cloud; the obstacle point cloud cost map determination unit 520 is also used to determine the obstacle point cloud cost map based on the weight of the obstacle point cloud and the position of the obstacle point cloud.

[0183] Figure 15 This is a structural block diagram of a visual point cloud processing apparatus according to another embodiment of this disclosure. Figure 15 As shown, the visual point cloud processing device may further include:

[0184] The obstacle point cloud cost extraction module 600 is used to extract obstacle point cloud cost information of the target area, including the initial position and the target position, from the obstacle point cloud cost map based on the initial position and the target position of the vehicle.

[0185] The trajectory planning module 700 is used to plan the driving trajectory of the vehicle based on the initial position, the target position and the obstacle point cloud cost information of the target area, and to determine the planned driving trajectory of the vehicle from the initial position to the target position.

[0186] It should be noted that the specific implementation of the visual point cloud processing device in this disclosure is similar to the specific implementation of the visual point cloud processing method in this disclosure. For details, please refer to the visual point cloud processing method section. To reduce redundancy, further details will not be provided.

[0187] Exemplary electronic devices

[0188] Below, for reference Figure 16 To describe an electronic device according to embodiments of this disclosure. For example... Figure 16 As shown, the electronic device includes one or more processors 10 and memory 20.

[0189] The processor 10 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0190] The memory 20 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the visual point cloud processing methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0191] In one example, the electronic device may also include an input device 30 and an output device 40, these components being interconnected via a bus system and / or other forms of connection mechanism (not shown). The input device 30 may be, for example, a keyboard, a mouse, etc. The output device 40 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, etc.

[0192] Of course, for the sake of simplicity, Figure 16 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0193] Exemplary computer-readable storage media

[0194] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0195] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0196] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0197] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0198] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0199] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0200] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0201] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for processing visual point clouds, comprising: Acquire image sequences captured by the vehicle-mounted camera within a preset time period; Based on the image sequence, the original point cloud data for the preset time period is determined, wherein the original point cloud data includes the location of the original point cloud and the image acquisition time; Based on the time difference between the image acquisition time and the target time of the original point cloud, the weight of the original point cloud is determined by the correspondence between the preset time difference and the weight, wherein the target time is the end acquisition time of the preset time period. Based on the position and weight of the original point cloud, point cloud densification is performed on the original point cloud data to obtain dense point cloud data. Based on the dense point cloud data, an obstacle point cloud cost map is determined at a distance of a preset range from the vehicle.

2. The method according to claim 1, wherein, The preset time period is a time period extending from the current time as the end time of data collection by a preset time length, and the weight of the original point cloud is inversely correlated with the time difference.

3. The method according to claim 2, wherein, The weights of the original point cloud exhibit either a linear inverse correlation with the time difference or an inverse correlation across different time difference stages.

4. The method according to claim 1, wherein, The process of performing point cloud densification on the original point cloud data based on the position and weight of the original point cloud to obtain the dense point cloud data includes: The pixel values ​​of the original point cloud are determined based on the weights of the original point cloud. Based on the position and pixel values ​​of the original point cloud, Gaussian blur processing is performed on the original point cloud to obtain blurred point cloud data. The Gaussian blur processing includes expanding the position range of the point cloud and adjusting the pixel values ​​of the point cloud. The blurred point cloud data includes the position and pixel values ​​of the blurred point cloud. Based on the position and pixel values ​​of the original point cloud and the position and pixel values ​​of the blurred point cloud, the original point cloud and the blurred point cloud data are overlaid to obtain the dense point cloud data.

5. The method according to any one of claims 1-4, wherein, The step of determining an obstacle point cloud cost map within a preset range near the vehicle based on the dense point cloud data includes: Determine the weight percentage of obstacle cost in the cost of trajectory planning for the vehicle; Based on the weighted proportion of the obstacle cost, the weight and position of the original point cloud, the obstacle point cloud cost map is determined, wherein the obstacle point cloud cost map includes the obstacle point cloud position and the cost of the vehicle traveling to the obstacle point cloud position.

6. The method according to claim 5, wherein, The process of determining the obstacle point cloud cost map based on the weighted proportion of the obstacle cost, the weight of the original point cloud, and its position includes: The weights of the obstacle point cloud are determined based on the weights of the original point cloud and the weight ratio of the obstacle cost. The location of the obstacle point cloud is determined based on the location of the original point cloud; Based on the weights and positions of the obstacle point clouds, a cost map of the obstacle point clouds is determined.

7. The method according to any one of claims 1-4, after determining the obstacle point cloud cost map within a preset range near the vehicle based on the dense point cloud data, further comprising: Based on the vehicle's initial position and target position, obstacle point cloud cost information of the target area, including the initial position and the target position, is extracted from the obstacle point cloud cost map; Based on the initial position, the target position, and the obstacle point cloud cost information of the target area, the vehicle's driving trajectory is planned to determine the planned driving trajectory of the vehicle from the initial position to the target position.

8. A visual point cloud processing apparatus, comprising: The image sequence acquisition module is used to acquire image sequences captured by the vehicle-mounted camera within a preset time period; The raw point cloud data determination module is used to determine the raw point cloud data of the preset time period based on the image sequence, wherein the raw point cloud data includes the location of the raw point cloud and the image acquisition time; The point cloud weight determination module is used to determine the weight of the original point cloud based on the time difference between the image acquisition time and the target time, and through a pre-set correspondence between the time difference and the weight, wherein the target time is the end acquisition time of the preset time period. The dense point cloud data acquisition module is used to perform point cloud densification processing on the original point cloud data based on the position and weight of the original point cloud to obtain dense point cloud data. The obstacle point cloud cost map determination module is used to determine the obstacle point cloud cost map within a preset range near the vehicle based on the dense point cloud data.

9. A computer-readable storage medium storing a computer program for performing the visual point cloud processing method according to any one of claims 1-7.

10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the visual point cloud processing method according to any one of claims 1-7.

Images