A point cloud enhancement method, apparatus, device and storage medium

Abstract

The application discloses a point cloud enhancement method, device, equipment and storage medium, the method comprises the steps of obtaining a first point cloud frame and a second point cloud frame scanned in the same scene; determining a target point cloud belonging to at least a first target object from the first point cloud frame, and determining a target interpolation position between a second target object pair in the second point cloud frame; based on the target point cloud, the second point cloud frame is enhanced at the target interpolation position to obtain an enhanced second point cloud frame. Through the above manner, the application can realize the enhancement of the point cloud.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a point cloud enhancement method, apparatus, device, and storage medium. Background Technology

[0002] Currently, target detection using 3D point clouds has been widely applied. For example, in the field of autonomous driving, using LiDAR to collect point cloud data for target detection is a crucial part of autonomous driving perception technology. During driving, rapid and efficient target detection can greatly ensure driving safety. While 3D point cloud target detection based on deep learning methods for training network models has also been widely adopted, training a reliable deep neural network model requires a large amount of diverse and accurately labeled 3D point cloud data. However, the cost of collecting, labeling, and maintaining this data is high, and the data's cross-task scalability is poor, becoming a major obstacle to training point cloud feature deep neural network models.

[0003] Therefore, it is of great significance to learn how to use existing point clouds for data augmentation. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a point cloud enhancement method, apparatus, device, and storage medium capable of enhancing point clouds.

[0005] To address the aforementioned technical problems, this application provides a point cloud enhancement method, which includes: acquiring a first point cloud frame and a second point cloud frame scanned in the same scene; determining a target point cloud belonging to at least a first target object from the first point cloud frame, and determining the target interpolation position located between a pair of second target objects from the determined second point cloud frame; and enhancing the second point cloud frame at the target interpolation position based on the target point cloud to obtain the enhanced second point cloud frame.

[0006] To address the aforementioned technical problems, another technical solution adopted in this application is to provide a point cloud enhancement device, which includes: an acquisition module for acquiring a first point cloud frame and a second point cloud frame scanned in the same scene; a determination module for determining, from the first point cloud frame, a target point cloud belonging to at least a first target object, and from the determined second point cloud frame, a target interpolation position located between a pair of second target objects; and an enhancement module for enhancing the second point cloud frame based on the target point cloud at the target interpolation position to obtain an enhanced second point cloud frame.

[0007] The step of determining the target interpolation position between the second target object pairs in the second point cloud frame includes: determining at least one set of second target object pairs from the second point cloud frame; for each set of second target object pairs, determining the target interpolation position between the second target object pairs based on several sets of point cloud pairs belonging to the second target object pairs, wherein the two points in the point cloud pair belong to different second target objects in the second target object pair respectively.

[0008] Among them, the second target object pair and the first target object are of the same object type.

[0009] Specifically, determining the target interpolation position between the two target object pairs based on several point cloud pairs belonging to the second target object pair includes: for each point cloud pair, obtaining the sum of distances from the candidate interpolation point to each point in the point cloud pair according to a preset point cloud enhancement ratio, and selecting the candidate interpolation point corresponding to the shortest sum of distances as the target interpolation point of the point cloud pair; and determining the target interpolation position based on the target interpolation point of each point cloud pair.

[0010] Among them, determining the target interpolation position based on the target interpolation points of each pair of point clouds includes: obtaining the average position of the target interpolation points of each pair of point clouds as the target interpolation position.

[0011] The step of determining at least one target point cloud belonging to the first target object from the first point cloud frame includes: performing polar coordinate transformation on the first point cloud frame to obtain a third point cloud frame; determining the positioning angle range of the first target object based on the coordinate information of each point in the third point cloud frame; and selecting at least one candidate point cloud belonging to the first target object within the positioning angle range of the third point cloud frame as the target point cloud.

[0012] The coordinate information of a point includes the point's positioning angle in the polar coordinate system. The positioning angle is the angle between the line segment and the horizontal axis on the horizontal plane. The line segment is the line connecting the projection point of the point on the horizontal plane and the origin of the coordinate system.

[0013] Before selecting candidate point clouds belonging to the first target object within the positioning angle range of at least the third point cloud frame as the target point cloud, the point cloud enhancement method further includes: obtaining the detection result of the background point cloud at the target interpolation position; wherein the detection result includes whether the background point cloud meets the preset conditions; selecting candidate point clouds belonging to the first target object within the positioning angle range of at least the third point cloud frame as the target point cloud includes: selecting candidate point clouds or selecting all point clouds related to the first target object within the positioning angle range as the target point cloud based on the detection result, wherein all point clouds related to the first target object include candidate point clouds.

[0014] The step of selecting candidate point clouds or all point clouds related to the first target object within the positioning angle range as target point clouds based on the detection results includes: selecting candidate point clouds as target point clouds in response to the detection results including background point clouds meeting preset conditions; and selecting all point clouds related to the first target object within the positioning angle range as target point clouds in response to the detection results including background point clouds not meeting preset conditions.

[0015] The preset conditions include: the overlap rate between the background point cloud at the target interpolation position and the background point cloud of the first target object is not less than a preset threshold.

[0016] The process of enhancing the second point cloud frame at the target interpolation position based on the target point cloud to obtain the enhanced second point cloud frame includes: transforming the target point cloud to obtain the processed target point cloud; and inserting the processed target point cloud into the target interpolation position to obtain the enhanced second point cloud frame.

[0017] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, including a memory and a processor coupled to each other, wherein the memory stores program instructions; and the processor is used to execute the program instructions stored in the memory to implement the above-mentioned method.

[0018] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing program instructions that can be executed to implement the above-mentioned method.

[0019] The above scheme, after acquiring a first point cloud frame and a second point cloud frame scanned in the same scene, determines at least one target point cloud belonging to a first target object from the first point cloud frame, and determines the target interpolation position located between the second target object pairs from the second point cloud frame. Then, based on the target point cloud, the second point cloud frame is enhanced at the target interpolation position to obtain an enhanced second point cloud frame. It is understood that this application's scheme can utilize two point cloud frames for data enhancement, achieving point cloud enhancement. Furthermore, compared to enhancing a point cloud using only a single frame, this application's method of enhancing point cloud using two frames improves the diversity of point cloud data. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating an embodiment of the point cloud enhancement method provided in this application;

[0021] Figure 2 This is the target interpolation position between the second target object pairs provided in this application;

[0022] Figure 3 yes Figure 1A partial flowchart of one embodiment of step S12 is shown below;

[0023] Figure 4 A schematic diagram of the positioning angle range of the third point cloud frame and the first target object provided in this application;

[0024] Figure 5 This is a schematic diagram of the target point cloud provided in this application;

[0025] Figure 6 This is a schematic diagram provided in this application of inserting a determined target point cloud into the target interpolation position to obtain an enhanced second point cloud frame;

[0026] Figure 7 yes Figure 1 The flowchart shown in step S12 is another part of an embodiment.

[0027] Figure 8 This is a schematic diagram of an embodiment of the point cloud processing method provided in this application;

[0028] Figure 9 This is a schematic diagram of the framework of an embodiment of the point cloud enhancement device provided in this application;

[0029] Figure 10 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application;

[0030] Figure 11 This is a schematic diagram of the structure of the computer-readable storage medium provided in this application. Detailed Implementation

[0031] To make the purpose, technical solution and effects of this application clearer and more explicit, the following describes this application in further detail with reference to the accompanying drawings and embodiments.

[0032] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0033] It should be noted that point cloud augmentation is a method to generate more point cloud data from limited point cloud data, thereby facilitating the training of network models using more point cloud data. This enables network models trained on a large amount of point cloud data to perform data processing (such as object detection) quickly and efficiently.

[0034] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the point cloud enhancement method provided in this application. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily replace it with a similar method. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, this embodiment includes:

[0035] S11: Obtain the first and second point cloud frames scanned in the same scene.

[0036] This embodiment uses a target point cloud determined from a first point cloud frame to enhance a second point cloud frame at the target interpolation position, thus obtaining an enhanced second point cloud frame and achieving point cloud enhancement. Point cloud enhancement involves generating more point cloud data from limited point cloud data; in this embodiment, the enhancement of the second point cloud frame is achieved using the target point cloud from the first point cloud frame.

[0037] It should be noted that in this embodiment, the interpolated point cloud (target point cloud) is determined from the first point cloud frame. Then, based on the interpolated point cloud, the second point cloud frame is enhanced at the target interpolation position to obtain the enhanced second point cloud frame. The number of second point cloud frames can be one frame or multiple frames, such as two, three, or four frames. That is, point cloud enhancement can be performed using the target point cloud at each target interpolation position in one second point cloud frame, or it can be performed using the target point cloud at each target interpolation position in multiple second point cloud frames. The specific number of second point cloud frames can be determined based on the actual enhancement effect, etc., and is not specifically limited here.

[0038] In this embodiment, the first point cloud frame and the second point cloud frame can be acquired using one or more point cloud acquisition devices such as LiDAR, millimeter-wave radar, or cameras. The first and second point cloud frames are several point cloud frames scanned in the same scene, such as an urban road or a rural road. For example, both the first and second point cloud frames are several point cloud frames scanned in an urban road. It is understood that the urban road example is used here only; the specific scene can be determined according to the purpose or use of point cloud enhancement, and is not specifically limited here.

[0039] S12: Determine the target point cloud that belongs at least to the first target object from the first point cloud frame, and determine the target interpolation position in the second point cloud frame that is located between the second target object pairs.

[0040] In this embodiment, the target point cloud is the interpolation data source determined from the first point cloud frame, and the target interpolation position is the position to be interpolated determined from the second point cloud frame. For example, if the target point cloud (e.g., A) is determined from the first point cloud frame, and the target interpolation positions (e.g., B and C respectively) are determined from the second point cloud frame, then A can be inserted into positions B and C in the second point cloud frame respectively.

[0041] In one embodiment, the first point cloud frame is marked with the location and attribute information of each first target object. The attribute information of the first target object may include, but is not limited to, the type information to which the first target object belongs, and may also include the size information of the first target object, etc. In this embodiment, target point clouds belonging to at least the first target objects can be determined from the first point cloud frame based on the location and attribute information of the first target objects marked in the first point cloud frame. The target point cloud may be, but is not limited to, the point cloud belonging to the first target object, and may also be all point clouds related to the first target object, such as the point cloud belonging to the first target object and the ground point cloud belonging to the surface where the first target object is located.

[0042] In one implementation scenario, the first and second point cloud frames are acquired using LiDAR. To reduce the computational cost of determining the target point cloud using the LiDAR coordinate system and thus reduce the complexity of chip computing power, the first point cloud frame can be transformed into polar coordinates. For example, all points in the first point cloud frame, or all points related to the first target object, can be transformed into polar coordinates to obtain the third point cloud frame. Then, based on the coordinate information of each point in the third point cloud frame, the positioning angle range of the first target object is determined, and at least one candidate point cloud belonging to the first target object within the positioning angle range of the third point cloud frame is selected as the target point cloud. Here, the positioning angle range is the angle range corresponding to the first target object in the polar coordinate system.

[0043] In this embodiment, the target interpolation position determined in the second point cloud frame is at least one. In one implementation, the target interpolation position is the position between two pairs of second target objects (two second target objects), and the specific number of target interpolation positions in the second point cloud frame can be determined according to the number of pairs of second target objects. For example, if there are two pairs of second target objects in the second point cloud frame, then there are two target interpolation positions, which are the positions between each pair of second target objects.

[0044] In one specific implementation, the two second target objects in the second target object pair are of the same object type as the first target object. For example, both the second target objects and the first target object in the second target object pair are vehicles. It is understood that in this implementation, at least the first target object (vehicle) in the first point cloud frame is inserted between the two vehicles in the second target object pair in the second point cloud frame. Of course, in other implementations, the two second target objects and the first target object in the second target object pair can be of different object types. For example, the two second target objects in the second target object pair are flower beds or trees, and the first target object is a vehicle. It is understood that in this implementation, at least the first target object (vehicle) in the first point cloud frame is inserted between the two flower beds or two trees, or one flower bed and one tree, in the second point cloud frame.

[0045] It should be noted that the proposed solution is based on a target point cloud that belongs to at least the first target object. The second point cloud frame is enhanced at the target interpolation position between two second target objects within the second point cloud frame, resulting in an enhanced second point cloud frame. In other words, the target point cloud to be inserted into the second point cloud frame is determined from the first point cloud frame. Therefore, when both the first and second point cloud frames are acquired using LiDAR, to reduce the computational cost of determining the target point cloud using the LiDAR coordinate system and thus reduce the complexity of chip computing power, only polar coordinate transformation can be performed on the first point cloud frame. After determining the target point cloud based on the polar coordinate transformed point cloud, to ensure consistency with the point cloud information of the second point cloud frame, the determined target point cloud can be transformed back to the original LiDAR coordinate system, facilitating the subsequent insertion of the target point cloud into the target interpolation position in the second point cloud frame. Of course, to ensure consistency with the point cloud information of the second point cloud frame, polar coordinate transformation can also be performed on the second point cloud frame, or the polar coordinate transformation of the first point cloud frame can be omitted. The specific choice depends on the actual enhancement effect of the point cloud, and no specific limitation is made here.

[0046] Furthermore, the proposed solution involves inserting target point clouds belonging at least to the first target object between pairs of second target objects in the second point cloud frame. However, the target interpolation position between the two second target objects in the second target object pair directly affects the interpolation position of the target point cloud belonging at least to the first target object. In other words, the target interpolation position between the pairs of second target objects determines the interpolation position of the target (target point cloud) to be inserted. It is understandable that if multiple pairs of second target objects of different types exist, and the target interpolation position is relatively arbitrary, it is easy to insert a first target object of type A (e.g., a vehicle) between pairs of second target objects of type B (e.g., two stones or two potted plants).

[0047] For example, please refer to Figure 2 , Figure 2 This is the target interpolation position between the second target object pair provided in this application. Figure 2 As can be seen, when a target is inserted at interpolation position C1 between targets A1 and B1, the sum of the distances from interpolation position C1 to targets A1 and B1 is the shortest. In this case, it is clear that a target was inserted at interpolation position C1 between A1 and B1, not between targets D1 and E1. However, when a target is inserted at interpolation position C2 between targets A2 and B2, the sum of the actual distances from interpolation position C2 to targets A2 and B2 is greater than the sum of the shortest distances between targets A2 and B2. Figure 2 As can be seen, when multiple pairs of second target objects exist, because the sum of the actual distances from the target interpolation position C2 to targets A2 and B2 is greater than the sum of the shortest distances between targets A2 and B2, it is easily mistaken for a target being inserted between targets D2 and E2, rather than between targets A2 and B2. It is understandable that the intended insertion point was between targets A2 and B2, but due to the influence of the distance from the target interpolation position C2 to the second target object pair (A2 and B2), the target was inserted between targets D2 and E2.

[0048] Therefore, in one embodiment, in order to avoid the above-mentioned target insertion error (inserting the target into the wrong second target object pair) caused by the randomness of the target interpolation position between the second target object pairs, the position corresponding to the sum of the shortest distances from the two second target objects in the second target object pair can be determined as the target interpolation position, so as to ensure that the selected target to be interpolated (target point cloud) can be inserted into the correct second target object pair.

[0049] For each pair of second target objects, the target interpolation position corresponding to the sum of the shortest distances can be determined according to the following formula:

[0050] S mix (λ)=(1-λ)·S A +λ·S B

[0051] y mix (λ)=(1-λ)·y A +λ·y B

[0052] λ∈(0,1)

[0053] In the above formula, S A S represents the location information of target A in the second target object pair. BThis represents the position information of target B in the second target object pair, λ represents the preset point cloud enhancement ratio, and S mix (λ) represents the location information of the target interpolation position corresponding to the sum of shortest distances when the preset point cloud enhancement ratio is λ. A larger λ value indicates that the target interpolation position corresponding to the sum of shortest distances is closer to target B, and vice versa. Furthermore, y A This represents the attribute information of target A in the second target object pair, y B This represents the attribute information of target B in the second target object pair, y mix (λ) Attribute information of the target interpolation position corresponding to the sum of the shortest distances.

[0054] In one embodiment, when determining the target interpolation position located between the second target object pairs in the second point cloud frame, at least one set of second target object pairs can be determined from the second point cloud frame first. Then, for each set of second target object pairs, the position between the center points of each second target object in the second target object pair can be directly used as the target interpolation position corresponding to the target point cloud.

[0055] Of course, in other embodiments, when determining the target interpolation position between the second target object pairs in the second point cloud frame, at least one set of second target object pairs can be determined from the second point cloud frame first. Then, for each set of second target object pairs, the target interpolation position between the second target object pairs can be determined based on several point cloud pairs belonging to the second target object pairs, wherein the two points in the point cloud pair belong to different second target objects in the second target object pair. For example, taking two vehicles as an example, the target interpolation position between the two vehicles can be determined based on several point cloud pairs at different positions of the two vehicles (e.g., the center point of each vehicle, the vertex of the left front side of the vehicle, the vertex of the right front side of the vehicle, the vertex of the left rear side of the vehicle, etc.).

[0056] It should be noted that, based on the positions of several points and / or the center point of the outer contours of the two second target objects in each pair of second target objects, a one-to-one correspondence can be established to form several pairs of point clouds belonging to that pair of second target objects. The two points in each pair of point clouds belong to different second target objects within that pair. For example, the four vertices of the outer contour of second target object A in a pair of second target objects are A1, A2, A3, and A4 (front left, front right, rear left, and rear right), and the four vertices of the outer contour of second target object B are B1, B2, B3, and B4 (front left, front right, rear left, and rear right). The resulting pairs of point clouds belonging to that pair of second target objects are A1B1, A2B2, A3B3, and A4B4. Therefore, the target interpolation position between second target objects A and B can be determined based on these four pairs of point clouds.

[0057] S13: Based on the target point cloud, enhance the second point cloud frame at the target interpolation position to obtain the enhanced second point cloud frame.

[0058] In one embodiment, the target point cloud determined from the first point cloud frame can be directly inserted into at least one target interpolation position in the second point cloud frame to obtain an enhanced second point cloud frame.

[0059] Of course, in other implementations, after determining the target point cloud from the first point cloud frame, the target point cloud can be transformed according to the actual scene in the second point cloud frame to obtain the processed target point cloud. Then, the processed target point cloud is inserted into the target interpolation position to obtain the enhanced second point cloud frame.

[0060] For example, to ensure that the size of the first target object corresponding to the target point cloud determined in the first point cloud frame is appropriate for the size of the second target object pair in the second point cloud frame, the target point cloud or the point cloud belonging to the first target object in the target point cloud can be resized (e.g., enlarged by 1.5 times or reduced in size according to a preset ratio), and the resized target point cloud can be inserted into the target interpolation position to obtain the enhanced second point cloud frame; or, to ensure that the first target object corresponding to the target point cloud determined in the first point cloud frame can adapt to the real scene corresponding to the second point cloud frame when inserted into the target interpolation position, the angle of the target point cloud determined in the first point cloud frame can be adjusted, and then the angle-adjusted target point cloud can be inserted into the target interpolation position to obtain the enhanced second point cloud frame. It should be noted that the method of transforming the target point cloud can be determined according to actual needs, and no specific limitation is made here.

[0061] The above scheme, after acquiring a first point cloud frame and a second point cloud frame scanned in the same scene, determines at least one target point cloud belonging to a first target object from the first point cloud frame, and determines the target interpolation position located between the second target object pairs from the second point cloud frame. Then, based on the target point cloud, the second point cloud frame is enhanced at the target interpolation position to obtain an enhanced second point cloud frame. It is understood that this embodiment can utilize two point cloud frames for data enhancement, achieving point cloud enhancement. Furthermore, compared to enhancing a point cloud using only a single frame, this embodiment's method of using two point cloud frames for point cloud enhancement improves the diversity of point cloud data.

[0062] It should be noted that in some implementation scenarios, such as during autonomous driving, after acquiring the first and second point cloud frames scanned from the same scene by a point cloud acquisition device (e.g., LiDAR or camera), target point clouds belonging to at least the first target object can be determined from the first point cloud frame, and the target interpolation position located between the second target object pairs in the second point cloud frame can be determined. Then, based on the target point cloud, the second point cloud frame is enhanced at the target interpolation position. It is understood that the point cloud enhancement scheme of this application can generate more point cloud data from limited point cloud data. In addition, since the scheme of this application uses the target point cloud belonging to at least the first target object in the first point cloud frame to enhance the point cloud data in the second point cloud frame, it can improve the diversity of the enhanced second point cloud frame data, so as to use more point cloud data (first point cloud frame, second point cloud frame, and enhanced second point cloud frame) to train the network model, and then use the trained network model for data processing (e.g., target detection).

[0063] Please see Figure 3 , Figure 3 yes Figure 1 The diagram shows a partial flowchart of one embodiment of step S12. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow the same pattern. Figure 3 The illustrated process sequence is limited. For example... Figure 3 As shown, in this embodiment, determining the target point cloud belonging to at least the first target object from the first point cloud frame includes:

[0064] S31: Perform polar coordinate transformation on the first point cloud frame to obtain the third point cloud frame.

[0065] This embodiment is used to determine the positioning angle range of the first target object based on the coordinate information of each point in the third point cloud frame after polar coordinate transformation, and to select at least the candidate point cloud belonging to the first target object within the positioning angle range as the target point cloud.

[0066] In this embodiment, the points in the first point cloud frame acquired by the lidar are transformed into polar coordinates to obtain a third point cloud frame containing all the points in a polar coordinate system. In one embodiment, the coordinate information of each point in the transformed third point cloud frame includes the positioning angle of the point in the polar coordinate system, wherein the positioning angle is the angle between the line segment formed by the projection point of the point on the horizontal plane and the origin of the coordinate system and the horizontal axis of the horizontal plane. Of course, the coordinate information of each point in the transformed third point cloud frame also includes the distance information between the point and the origin of the coordinate system, as well as the angle information between the half-plane formed by the vertical axis and the point and the coordinate plane (the plane formed by the vertical axis and the horizontal axis).

[0067] S32: Determine the positioning angle range of the first target object based on the coordinate information of each point in the third point cloud frame.

[0068] The positioning angle range of the first target object is the angle range containing the first target object determined within the entire third point cloud frame. For an example, please refer to [link to example]. Figure 4 , Figure 4 A schematic diagram illustrating the positioning angle range of the third point cloud frame and the first target object provided in this application. (See diagram below.) Figure 4 As shown, the positioning angle range of the first target object is the angle range containing the first target object determined from the third point cloud frame.

[0069] In this embodiment, the third point cloud frame contains the coordinate information of each point in the polar coordinate system. For each point in the polar coordinate system, its coordinate information includes its positioning angle in the polar coordinate system. Therefore, the range of positioning angles belonging to each point of the first target object can be determined based on the positioning angle information of each point. Of course, in other embodiments, the range of positioning angles belonging to each point of the first target object can also be determined based on the distance information between each point in the third point cloud frame and the origin of the polar coordinate system, or based on the angle between the half-plane formed by the vertical axis and the point and the coordinate plane (the plane formed by the vertical axis and the horizontal axis).

[0070] S33: Select at least one candidate point cloud belonging to the first target object within the positioning angle range of the third point cloud frame as the target point cloud.

[0071] In one embodiment, before step S33, the detection result of the background point cloud at the target interpolation position can be obtained first, and then step S33 is executed based on the detection result of the background point cloud at the target interpolation position. In a specific implementation scenario, step S33 includes: based on the detection result of the background point cloud at the target interpolation position, candidate point clouds belonging to the first target object within the positioning angle range in the third point cloud frame or all point clouds related to the first target object within the positioning angle range can be selected as the target point cloud. The all point clouds related to the first target object within the positioning angle range include candidate point clouds belonging to the first target object.

[0072] For example, please refer to Figure 5 , Figure 5 This is a schematic diagram of the target point cloud provided in this application. For example... Figure 5 As shown, based on the detection results of the background point cloud at the target interpolation location, it can be determined whether to select all point clouds related to the first target object within the positioning angle range, or candidate point clouds belonging to the first target object, as the target point cloud. It should be noted that the candidate point cloud belonging to the first target object is the point cloud belonging to the first target object; all point clouds related to the first target object within the positioning angle range include not only the point cloud belonging to the first target object, but also the background point cloud of the first target object. The background point cloud of the first target object can be, but is not limited to, ground point clouds belonging to the ground at the location of the first target object, or it can be obstacle points (such as station signs or trees) at the location of the first target object within the positioning angle range. The specific background point cloud of the first target object can be determined based on the real scene of the first point cloud frame, and no specific limitation is made here.

[0073] In some embodiments, the detection result of the background point cloud at the target interpolation location includes whether the background point cloud meets a preset condition. The preset condition can be determined according to the actual scenario. For example, in a specific implementation scenario, the preset condition is that the overlap rate between the background point cloud at the target interpolation location and the background point cloud of the first target object is not less than a preset threshold. That is, the detection result of the background point cloud at the target interpolation location can be determined based on whether the overlap rate between the background point cloud at the target interpolation location and the background point cloud of the first target object is not less than a preset threshold. Specifically, if the overlap rate between the background point cloud at the target interpolation position and the background point cloud of the first target object is not lower than a preset threshold, then the detection result of the background point cloud at the corresponding target interpolation position meets the preset condition. In this case, it indicates that there are many background point clouds at the target interpolation position, so only the candidate point cloud belonging to the first target object is selected as the target point cloud to reduce the impact of the background point cloud of the first target object on the point cloud enhancement effect of the second point cloud frame. If the overlap rate between the background point cloud at the target interpolation position and the background point cloud of the first target object is lower than the preset threshold, then the detection result of the background point cloud at the corresponding target interpolation position does not meet the preset condition. In this case, it indicates that there are few background point clouds at the target interpolation position, so all point clouds related to the first target object within the positioning angle range are selected as the target point cloud. This ensures that after inserting the target point cloud, the target point cloud at the target interpolation position of the second point cloud frame contains not only the point cloud belonging to the first target object but also the background point cloud of the first target object, making the scene at the target interpolation position of the second point cloud frame more consistent with the real scene.

[0074] Please see Figure 6 , Figure 6This is a schematic diagram provided in this application of inserting a determined target point cloud into the target interpolation position to obtain an enhanced second point cloud frame. For example... Figure 6 As shown, in a specific implementation scenario, after determining the target point cloud (e.g., target point cloud 1 including all point clouds related to the first target object and / or target point cloud 2 belonging to the first target object) from the first point cloud frame, and after determining the target interpolation position (which may be multiple) in the second point cloud frame, it can be determined whether to insert target point cloud 1 or target point cloud 2 as the target point cloud at each target interpolation position based on the detection results of the background point cloud at the target interpolation position in the second point cloud frame. Specifically, based on the size of the second target object pair corresponding to the target interpolation position and the corresponding real scene, target point cloud 1 or target point cloud 2 can be subjected to size transformation, angle adjustment, or other processing before being inserted into each target interpolation position. It should be noted that... Figure 6 The white box on the right shows the diagram after the processed target point cloud 1 or target point cloud 2 is inserted into each target interpolation position.

[0075] The following formula represents the target interpolation position where the target point cloud is inserted into the second point cloud frame:

[0076]

[0077]

[0078] In the formula, s represents the first point cloud frame, s' represents the second point cloud frame, M represents the target point cloud determined from the first point cloud frame s, ⊙ represents the element combination and splicing operator, and R ω This indicates a target point cloud copying operation (i.e., copying (inserting) the processed target point cloud from the first point cloud frame to the target interpolation position in the second point cloud frame), where C and Ω represent the processing logic, which can be determined according to the actual situation, and y s This represents the annotation information of the first point cloud frame, y s’ This represents the annotation information of the second point cloud frame; the above formula indicates that the target point cloud M is selected from the first point cloud frame s, and then the processed target point cloud M is inserted into the second point cloud frame s. ’ middle.

[0079] Please see Figure 7 , Figure 7 yes Figure 1 The illustrated step S12 is a partial flowchart of another embodiment. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow the same pattern. Figure 7 The illustrated process sequence is limited. For example... Figure 7 As shown, in this embodiment, determining the target interpolation position located between the second target object pairs in the second point cloud frame includes:

[0080] S71: Determine at least one pair of second target objects from the second point cloud frame.

[0081] This embodiment is used to determine the target interpolation position of the target point cloud in the second point cloud frame.

[0082] In this embodiment, the number of second target object pairs determined from the second point cloud frame is at least one set. The specific number of second target object pairs can be determined based on the actual scene conditions in the second point cloud frame (e.g., the distance or type between two second target objects) or the application scenario of point cloud enhancement. Furthermore, the two second target objects in each set of second target object pairs can be the same or different types of objects. The specific number of second target object pairs and the object types of the two second target objects in each pair are not specifically limited here.

[0083] S72: For each pair of second target objects, based on several pairs of point clouds belonging to the pair of second target objects, determine the target interpolation position between the pairs of second target objects, wherein the two points in the point cloud pair belong to different second target objects in the pair of second target objects.

[0084] In this embodiment, after determining each group of second target object pairs, for each group of second target object pairs, the target interpolation position located between the second target object pairs can be determined based on several point cloud pairs belonging to the second target object pairs.

[0085] Specifically, for each pair of point clouds between the second target object pairs, the candidate interpolation points corresponding to the sum of the shortest distances can be used as the target interpolation points for the point cloud pairs according to a preset point cloud enhancement ratio. The target interpolation position is then determined based on the target interpolation points of each pair of point clouds. It is understood that the target interpolation position determined based on the target interpolation points corresponding to the sum of the shortest distances between each pair of point clouds has the shortest sum of distances to the second target object pair. Therefore, the target interpolation position determined according to this embodiment can accurately insert the target to be interpolated (target point cloud) into a suitable position between the second target object pairs.

[0086] For example, after determining each pair of point cloud points, for each pair, according to a preset point cloud enhancement ratio, the sum of distances from candidate interpolation points to each point in the point cloud pair can be obtained first, and the candidate interpolation point corresponding to the shortest sum of distances can be selected as the target interpolation point for the point cloud pair. Then, based on the target interpolation points of each pair of point cloud points, the target interpolation position can be determined. In some embodiments, a relevant model or algorithm can be used to obtain the sum of distances from candidate interpolation points to each point in the point cloud pair. For example, the EMD distance model can be used to obtain the sum of distances from candidate interpolation points to each point in the point cloud pair, as well as the candidate interpolation point corresponding to the shortest sum of distances.

[0087] First, each pair of point clouds can be determined. For example, the following formula can be used as a reference:

[0088]

[0089] In the formula, x i Let y represent several points in target A of the second target object pair. j ξ represents several points in target B that correspond to target A in the second target object pair. * This represents a set of point cloud pairs representing the second target object pair (a set of one-to-one correspondences between points in target A and target B).

[0090] The function corresponding to the sum of the shortest distances can be set as follows: In the formula, λ∈(0,1) represents the point cloud enhancement ratio. Let x represent the candidate interpolation point (target interpolation point of the i-th point cloud pair) corresponding to the sum of the shortest distances between the two points in the i-th point cloud pair, where x i , Let represent two points in the i-th point cloud pair. It should be noted that the function corresponding to the sum of the shortest distances can generate target interpolation point sets for several point cloud pairs. Then, based on the target interpolation points of each pair of point clouds, the target interpolation position is determined.

[0091] In a specific implementation scenario, after determining the target interpolation points between each pair of point cloud points, the average position of the target interpolation points for each pair can be obtained, and this average position is used as the target interpolation position. For example, this can be achieved using... The average position of the target interpolation points for each pair of point clouds is used as the target interpolation position. Here, N represents the number of point cloud pairs. Let i represent the target interpolation point of the i-th point cloud pair, where i ∈ (1, 2...N).

[0092] In one specific embodiment, please refer to Figure 8 , Figure 8This is a schematic diagram of an embodiment of the point cloud processing method provided in this application. Figure 8 As shown, the point cloud within the positioning angle range of the first point cloud frame (or the third point cloud frame after polar coordinate transformation of the first point cloud frame) is taken as the target point cloud. The target interpolation position in the second point cloud frame is cropped, and then the target point cloud within the positioning angle range of the first point cloud frame is inserted into the target interpolation position of the second point cloud frame to obtain the enhanced second point cloud frame. The specific enhancement operation is as follows:

[0093]

[0094]

[0095] In the formula, s A Indicates the first point cloud frame, s B This represents the second point cloud frame. This represents the target point cloud located within the angular range (α, β) from the first point cloud frame. The operator represents the combination and concatenation of elements; y A This represents the annotation information of the first point cloud frame, y B The above formula represents the annotation information of the second point cloud frame. It means that the target point cloud within the positioning angle range (α,β) of the first point cloud frame is cut and inserted into the target interpolation position of the second point cloud frame.

[0096] Please see Figure 9 , Figure 9 This is a schematic diagram of a framework of an embodiment of the point cloud enhancement device provided in this application. In this embodiment, the point cloud enhancement device 90 includes an acquisition module 91, a determination module 92, and an enhancement module 93. The acquisition module 91 is used to acquire a first point cloud frame and a second point cloud frame scanned in the same scene; the determination module 92 is used to determine, from the first point cloud frame, a target point cloud belonging to at least a first target object, and to determine the target interpolation position in the second point cloud frame located between a pair of second target objects; the enhancement module 93 is used to enhance the second point cloud frame based on the target point cloud at the target interpolation position to obtain an enhanced second point cloud frame.

[0097] In some embodiments, the determining module 92 determines the target interpolation position in the second point cloud frame located between the second target object pairs, including: determining at least one set of second target object pairs from the second point cloud frame; for each set of second target object pairs, determining the target interpolation position between the second target object pairs based on several sets of point cloud pairs belonging to the second target object pairs, wherein the two points in the point cloud pair belong to different second target objects in the second target object pair respectively.

[0098] In some embodiments, the second target object pair and the first target object are of the same object type.

[0099] In some embodiments, the determining module 92 determines the target interpolation position between the second target object pairs based on several point cloud pairs belonging to the second target object pair, including: for each point cloud pair, obtaining the sum of distances from the candidate interpolation point to each point in the point cloud pair according to a preset point cloud enhancement ratio, and selecting the candidate interpolation point corresponding to the shortest sum of distances as the target interpolation point of the point cloud pair; and determining the target interpolation position based on the target interpolation point of each point cloud pair.

[0100] In some embodiments, the determining module 92 determines the target interpolation position based on the target interpolation points of each pair of point clouds, including: obtaining the average position of the target interpolation points of each pair of point clouds as the target interpolation position.

[0101] In some embodiments, the determining module 92 determines at least one target point cloud belonging to the first target object from the first point cloud frame, including: performing polar coordinate transformation on the first point cloud frame to obtain a third point cloud frame; determining the positioning angle range of the first target object based on the coordinate information of each point in the third point cloud frame; and selecting at least one candidate point cloud belonging to the first target object within the positioning angle range of the third point cloud frame as the target point cloud.

[0102] In some embodiments, the coordinate information of a point includes the positioning angle of the point in the polar coordinate system, where the positioning angle is the angle between the line segment and the horizontal axis on the horizontal plane, and the line segment is the line connecting the projection point of the point on the horizontal plane and the origin of the coordinate system.

[0103] In some embodiments, before the determining module 92 selects at least the candidate point cloud belonging to the first target object within the positioning angle range of the third point cloud frame as the target point cloud, the point cloud enhancement method further includes: obtaining the detection result of the background point cloud at the target interpolation position; wherein the detection result includes whether the background point cloud meets the preset conditions; selecting at least the candidate point cloud belonging to the first target object within the positioning angle range of the third point cloud frame as the target point cloud includes: selecting the candidate point cloud or selecting all point clouds related to the first target object within the positioning angle range as the target point cloud based on the detection result, wherein all point clouds related to the first target object include the candidate point cloud.

[0104] In some embodiments, the determining module 92 selects candidate point clouds or all point clouds related to the first target object within the positioning angle range as target point clouds based on the detection results, including: selecting candidate point clouds as target point clouds in response to the detection results including background point clouds meeting preset conditions; and selecting all point clouds related to the first target object within the positioning angle range as target point clouds in response to the detection results including background point clouds not meeting preset conditions.

[0105] In some embodiments, the preset conditions include: the overlap rate between the background point cloud at the target interpolation location and the background point cloud of the first target object is not less than a preset threshold.

[0106] In some embodiments, the enhancement module 93 enhances the second point cloud frame at the target interpolation position based on the target point cloud to obtain the enhanced second point cloud frame, including: transforming the target point cloud to obtain the processed target point cloud; and inserting the processed target point cloud into the target interpolation position to obtain the enhanced second point cloud frame.

[0107] Please see Figure 10 , Figure 10 This is a schematic diagram of an embodiment of the electronic device provided in this application. In this embodiment, the electronic device 100 includes a processor 101 and a memory 102.

[0108] Processor 101 can also be referred to as CPU (Central Processing Unit). Processor 101 may be an integrated circuit chip with signal processing capabilities. Processor 101 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 101 can be any conventional processor 101, etc.

[0109] The memory 102 in the electronic device 100 is used to store the program instructions required for the processor 101 to run.

[0110] The processor 101 is used to execute program instructions to implement the methods provided in any of the above embodiments and any non-conflicting combinations thereof.

[0111] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of the computer-readable storage medium provided in this application. The computer-readable storage medium 110 of this application embodiment stores program instructions 111, which, when executed, implement the methods provided in any of the above embodiments and any non-conflicting combinations. The program instructions 111 can form a program file and be stored in the computer-readable storage medium 110 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 110 includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

[0112] The above scheme, after acquiring a first point cloud frame and a second point cloud frame scanned in the same scene, determines at least one target point cloud belonging to a first target object from the first point cloud frame, and determines the target interpolation position located between the second target object pairs from the second point cloud frame. Then, based on the target point cloud, the second point cloud frame is enhanced at the target interpolation position to obtain an enhanced second point cloud frame. It is understood that this application's scheme can utilize two point cloud frames for data enhancement, achieving point cloud enhancement. Furthermore, compared to enhancing a point cloud using only a single frame, this application's method of enhancing point cloud using two frames improves the diversity of point cloud data.

[0113] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0114] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0116] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0118] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A point cloud enhancement method, characterized in that, The method includes: Acquire the first and second point cloud frames scanned in the same scene; Determine at least the target point cloud belonging to the first target object from the first point cloud frame, and determine the target interpolation position located between the second target object pairs in the second point cloud frame; Based on the target point cloud, the second point cloud frame is enhanced at the target interpolation position to obtain the enhanced second point cloud frame; Determining the target interpolation position in the second point cloud frame located between the second target object pairs includes: Determine at least one pair of the second target objects from the second point cloud frame; For each pair of second target objects, based on several pairs of point cloud belonging to the pair of second target objects, the target interpolation position between the pairs of second target objects is determined, wherein the two points in the point cloud pair belong to different second target objects in the pair of second target objects respectively; The step of determining the target interpolation position between the second target object pairs based on several point cloud pairs belonging to the second target object pair includes: For each pair of point clouds, according to the preset point cloud enhancement ratio, the sum of the distances from the candidate interpolation point to each point in the point cloud pair is obtained, and the candidate interpolation point corresponding to the shortest sum of the distances is selected as the target interpolation point of the point cloud pair. The target interpolation position is determined based on the target interpolation points of each group of point cloud pairs.

2. The method according to claim 1, characterized in that, The second target object and the first target object belong to the same object type.

3. The method according to claim 1, characterized in that, Determining the target interpolation position based on the target interpolation points of each group of point cloud pairs includes: The average position of the target interpolation points of each pair of point clouds is obtained as the target interpolation position.

4. The method according to claim 1, characterized in that, Determining at least one target point cloud belonging to the first target object from the first point cloud frame includes: The first point cloud frame is transformed into polar coordinates to obtain the third point cloud frame; Based on the coordinate information of each point in the third point cloud frame, the positioning angle range of the first target object is determined; At least one candidate point cloud belonging to the first target object within the positioning angle range of the third point cloud frame is selected as the target point cloud.

5. The method according to claim 4, characterized in that, The coordinate information of the point includes the positioning angle of the point in the polar coordinate system. The positioning angle is the angle between the line segment and the horizontal axis on the horizontal plane. The line segment is the line connecting the projection point of the point on the horizontal plane and the origin of the coordinate system.

6. The method according to claim 4, characterized in that, Before selecting candidate point clouds belonging to the first target object within the positioning angle range of at least the third point cloud frame as the target point cloud, the method further includes: Obtain the detection result of the background point cloud at the target interpolation position; wherein, the detection result includes whether the background point cloud meets the preset conditions; The step of selecting at least one candidate point cloud belonging to the first target object within the positioning angle range of the third point cloud frame as the target point cloud includes: Based on the detection results, the candidate point cloud or all point clouds related to the first target object within the positioning angle range are selected as the target point cloud, wherein all point clouds related to the first target object include the candidate point cloud.

7. The method according to claim 6, characterized in that, The step of selecting the candidate point cloud or all point clouds related to the first target object within the positioning angle range as the target point cloud based on the detection results includes: In response to the detection result including the background point cloud satisfying the preset condition, the candidate point cloud is selected as the target point cloud; In response to the detection result including the background point cloud not meeting the preset condition, all point clouds related to the first target object within the positioning angle range are selected as the target point cloud.

8. The method according to claim 6 or 7, characterized in that, The preset conditions include: the overlap rate between the background point cloud at the target interpolation position and the background point cloud of the first target object is not less than a preset threshold.

9. The method according to claim 1, characterized in that, The step of enhancing the second point cloud frame at the target interpolation position based on the target point cloud to obtain the enhanced second point cloud frame includes: The target point cloud is transformed to obtain the processed target point cloud; The processed target point cloud is inserted into the target interpolation position to obtain the enhanced second point cloud frame.

10. A point cloud enhancement device, characterized in that, The device includes: The acquisition module is used to acquire the first and second point cloud frames scanned in the same scene. The determining module is used to determine, from the first point cloud frame, at least the target point cloud belonging to the first target object, and to determine the target interpolation position in the second point cloud frame located between the second target object pairs; An enhancement module is used to enhance the second point cloud frame at the target interpolation position based on the target point cloud to obtain an enhanced second point cloud frame; Determining the target interpolation position in the second point cloud frame located between the second target object pairs includes: Determine at least one pair of the second target objects from the second point cloud frame; For each pair of second target objects, based on several pairs of point cloud belonging to the pair of second target objects, the target interpolation position between the pairs of second target objects is determined, wherein the two points in the point cloud pair belong to different second target objects in the pair of second target objects respectively; The step of determining the target interpolation position between the second target object pairs based on several point cloud pairs belonging to the second target object pair includes: For each pair of point clouds, according to the preset point cloud enhancement ratio, the sum of the distances from the candidate interpolation point to each point in the point cloud pair is obtained, and the candidate interpolation point corresponding to the shortest sum of the distances is selected as the target interpolation point of the point cloud pair. The target interpolation position is determined based on the target interpolation points of each group of point cloud pairs.

11. An electronic device, characterized in that, Including interconnected memory and processor, The memory stores program instructions; The processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed by a processor to implement the method of any one of claims 1-9.

Images