Point cloud coloring method, terminal and storage medium
By projecting and dedistorting the correspondence relationship between data points and pixel points after matching the point cloud with the image, the problem of lack of color information in traditional point cloud annotation is solved, and the recognition rate and labeling efficiency of the target object are improved.
Patent Information
- Application Number
- CN202311235484.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-22
- Publication Date
- 2025-04-08
AI Technical Summary
Due to the lack of color information during traditional point cloud labeling, the target object is not easily recognized and the labeling efficiency is low.
By acquiring the original point cloud and image, the target image is matched based on the first generation time of the point cloud, the corresponding relationship between the data points and the pixel points is projected and dedistorted, and finally the point cloud is colored.
This makes the target objects clearer in the point cloud, improves the recognition rate of the target objects, and thus improves the efficiency of the point cloud when manually labeling.
Smart Images
Figure CN119784984A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of computer technology, and in particular, relates to a point cloud coloring method, a terminal and a storage medium. Background Art
[0002] With the advancement of computer, chip and artificial intelligence technology, autonomous driving technology has also developed rapidly. Vehicles achieve autonomous driving through various models running in the perception module. These models are often trained through supervised learning, so a large amount of sample data from real scenes is required.
[0003] In related technologies, point clouds collected by LiDAR are often used as sample data. For example, lane lines, road signs, and obstacles in the point cloud are annotated to obtain sample data. However, since the point cloud collected by LiDAR only contains information such as position information and reflection intensity, and lacks color information, the target objects in it are not easy to identify, resulting in low annotation efficiency during manual annotation. Summary of the invention
[0004] The purpose of the present application is to provide a point cloud coloring method, a terminal and a storage medium, aiming to solve the problem that target objects are difficult to identify and the labeling efficiency is low when traditionally labeling point clouds.
[0005] A first aspect of an embodiment of the present application provides a point cloud coloring method, the method comprising:
[0006] Get multiple raw point clouds and multiple images;
[0007] For any original point cloud, based on the first generation time of the original point cloud, determining a target image matching the original point cloud from a plurality of images;
[0008] Projecting the original point cloud onto the target image that matches the original point cloud to obtain a correspondence between data points of the original point cloud and pixel points of the target image;
[0009] Based on the correspondence between the data points and the pixel points, the original point cloud is colored to obtain a colored point cloud.
[0010] In some embodiments, projecting the original point cloud onto the target image that matches the original point cloud to obtain a correspondence between data points of the original point cloud and pixel points of the target image includes:
[0011] Mapping the data points of the original point cloud from a first coordinate system to a second coordinate system, wherein the first coordinate system is a three-dimensional coordinate system corresponding to a point cloud acquisition device that acquires the original point cloud, and the second coordinate system is a three-dimensional coordinate system corresponding to an image acquisition device that acquires the target image;
[0012] Based on the device parameters of the image acquisition device, the original point cloud in the second coordinate system is projected onto the plane where the target image matched by the original point cloud is located, so as to obtain the correspondence between the data points of the original point cloud and the pixel points of the target image.
[0013] In some embodiments, projecting the original point cloud in the second coordinate system onto a plane where a target image matched with the original point cloud is located based on the device parameters of the image acquisition device to obtain a correspondence between data points of the original point cloud and pixel points of the target image includes:
[0014] determining a viewing cone of the image acquisition device based on the device parameters;
[0015] Deleting data points of the original point cloud outside the viewing cone in the second coordinate system to obtain a valid point cloud;
[0016] The data points of the valid point cloud in the second coordinate system are projected onto the plane where the target image matched by the original point cloud is located, so as to obtain the corresponding relationship between the data points of the original point cloud and the pixel points of the target image.
[0017] In some embodiments, deleting data points of the original point cloud outside the viewing cone in the second coordinate system to obtain a valid point cloud includes:
[0018] Determining positions of a plurality of planes of the viewing cone in the second coordinate system;
[0019] Based on the coordinates of the data points in the original point cloud in the second coordinate system and the positions of the multiple planes in the second coordinate system, the data points outside the range of the multiple planes of the viewing cone are deleted to obtain the valid point cloud.
[0020] In some embodiments, coloring the original point cloud based on the correspondence between the data points and the pixel points to obtain a colored point cloud includes:
[0021] For any data point in the original point cloud, based on the correspondence between the data point and the pixel point, determine a pixel point that matches the coordinates of the data point;
[0022] Assigning the color value of the pixel point to the matched data point to obtain a colored data point matched by the data point;
[0023] The colored point cloud is composed of colored data points that match multiple data points of the original point cloud.
[0024] In some embodiments, before projecting the original point cloud onto the target image matching the original point cloud to obtain the correspondence between the data points of the original point cloud and the pixel points of the target image, the method further includes:
[0025] Determining a moving speed and direction of a point cloud acquisition device that acquires the original point cloud;
[0026] The original point cloud is dedistorted based on the moving speed and direction.
[0027] In some embodiments, determining, based on the first generation time of the original point cloud, a target image matching the original point cloud from the plurality of images comprises:
[0028] determining a second generation time of each of the images;
[0029] A target image whose second generation time matches the first generation time is determined from the plurality of images.
[0030] In some embodiments, the method further comprises:
[0031] Determining sample data based on the colored point cloud;
[0032] Model training is performed based on the sample data.
[0033] A second aspect of an embodiment of the present application provides a point cloud coloring device, the device comprising:
[0034] An acquisition unit, used for acquiring a plurality of original point clouds and a plurality of images;
[0035] A first determining unit is configured to determine, for any original point cloud, a target image that matches the original point cloud from a plurality of images based on a first generation time of the original point cloud;
[0036] A projection unit, used to project the original point cloud onto the target image that matches the original point cloud, to obtain a correspondence between data points of the original point cloud and pixel points of the target image;
[0037] A coloring unit is used to color the original point cloud based on the correspondence between the data points and the pixel points to obtain a colored point cloud.
[0038] In some embodiments, the projection unit is used to map the data points of the original point cloud from a first coordinate system to a second coordinate system, where the first coordinate system is a three-dimensional coordinate system corresponding to a point cloud acquisition device that acquires the original point cloud, and the second coordinate system is a three-dimensional coordinate system corresponding to an image acquisition device that acquires the target image; based on the device parameters of the image acquisition device, the original point cloud in the second coordinate system is projected to the plane where the target image matching the original point cloud is located, to obtain the correspondence between the data points of the original point cloud and the pixel points of the target image.
[0039] In some embodiments, the projection unit is used to determine the viewing cone of the image acquisition device based on the device parameters; delete the data points of the original point cloud outside the viewing cone in the second coordinate system to obtain a valid point cloud; project the data points of the valid point cloud in the second coordinate system to the plane where the target image matched by the original point cloud is located, to obtain the correspondence between the data points of the original point cloud and the pixel points of the target image.
[0040] In some embodiments, the projection unit is used to determine the positions of multiple planes of the viewing cone in the second coordinate system; based on the coordinates of the data points in the original point cloud in the second coordinate system and the positions of the multiple planes in the second coordinate system, the data points outside the range of the multiple planes of the viewing cone are deleted to obtain the valid point cloud.
[0041] In some embodiments, the shading unit is used to determine, for any data point in the original point cloud, a pixel point that matches the coordinates of the data point based on the correspondence between the data point and the pixel point; assign the color value of the pixel point to the matched data point to obtain a colored data point that matches the data point; and compose the colored point cloud with the colored data points that match multiple data points of the original point cloud.
[0042] In some embodiments, the apparatus further comprises:
[0043] A first determining unit, used to determine a moving speed and direction of a point cloud acquisition device that acquires the original point cloud;
[0044] A dedistortion unit is used to perform dedistortion processing on the original point cloud based on the moving speed and direction.
[0045] In some embodiments, the first determining unit is used to determine the second generation time of each of the images; and to determine, from the plurality of images, a target image whose second generation time matches the first generation time.
[0046] In some embodiments, the apparatus further comprises:
[0047] A third determining unit, configured to determine sample data based on the colored point cloud;
[0048] A model training unit is used to perform model training based on the sample data.
[0049] A third aspect of an embodiment of the present application provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the point cloud coloring method as described above when executing the computer program.
[0050] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the point cloud coloring method as described above is implemented.
[0051] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: the data points in the original point cloud are colored by the correspondence between the data points in the original point cloud and the pixel points of the image. This scheme makes up for the lack of color information in the point cloud, making the target object represented in the point cloud clearer, so that the target object in the point cloud is easy to identify, thereby improving the efficiency of manual annotation of the point cloud. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 A schematic diagram of a point cloud coloring system provided by an exemplary embodiment is shown;
[0053] Figure 2 A schematic diagram of a process of a point cloud coloring method provided by the present application is shown;
[0054] Figure 3 A schematic diagram of a process of a point cloud coloring method provided by the present application is shown;
[0055] Figure 4 A schematic diagram of a process of a point cloud coloring method provided by the present application is shown;
[0056] Figure 5 A schematic diagram of the structure of a point cloud coloring device provided by the present application is shown;
[0057] Figure 6 It is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0058] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0059] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.
[0060] It should be understood that the orientation or position relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0061] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0062] The terms used in this application are explained below.
[0063] Point cloud: is a collection of data points in a certain coordinate system. The data points contain rich information, such as the three-dimensional coordinates, reflectivity, intensity value, time, etc. of the data points.
[0064] Please refer to Figure 1 , which shows a schematic diagram of a point cloud coloring system provided by an exemplary embodiment. The system includes: a terminal 10, an image acquisition device 20 and a point cloud acquisition device 30. The terminal 10 is respectively connected to the image acquisition device 20 and the point cloud acquisition device 30 for communication.
[0065] The terminal 10 is used to receive the image sent by the image acquisition device 20 and the point cloud sent by the point cloud acquisition device 30, and color the point cloud through the image. The terminal 10 is any terminal 10 with a data processing function. For example, the terminal 10 is a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and other terminal 10 devices. The embodiment of the present application does not impose any restrictions on the specific type of the terminal 10 device.
[0066] The image acquisition device 20 is used to acquire an image of the current environment at a first acquisition frequency and send the acquired image to the terminal 10. In some embodiments, the image acquisition device 20 is set on a vehicle, and acquires images at the first acquisition frequency while the vehicle is moving. The image acquisition device 20 is any device with an image acquisition function, for example, the image acquisition device 20 is a camera. The first acquisition frequency can be set as needed. For example, the first acquisition frequency is 10 Hz.
[0067] The point cloud acquisition device 30 is used to acquire the point cloud of the current environment at a second acquisition frequency and send the acquired point cloud to the terminal 10. In some embodiments, the point cloud acquisition device 30 is set on a vehicle, and acquires images at the second acquisition frequency while the vehicle is moving. The point cloud acquisition device 30 is any device with a point cloud acquisition function, for example, the point cloud acquisition device 30 is a laser radar. The second acquisition frequency can be set as needed. For example, the second acquisition frequency is any frequency value between 25 and 30 Hz.
[0068] It should be noted that the image acquisition device 20 and the point cloud acquisition device 30 can be set on the same vehicle or on different vehicles. In the embodiments of the present application, no specific limitation is made to this.
[0069] The image acquisition device 20 and the point cloud acquisition device 30 are used to collect current environmental data. For example, the image acquisition device 20 and the point cloud acquisition device 30 are used to collect environmental data such as road lane lines, road signs and obstacles. The collected environmental data is used as sample data used in the supervised learning process. In order to improve the efficiency of sample annotation, the present application proposes a point cloud coloring method, which makes the target objects in the point cloud easy to identify, thereby improving the efficiency of manual annotation of the point cloud.
[0070] It should be noted that the image acquisition device 20 and the point cloud acquisition device 30 may be modules with corresponding functions integrated on the terminal 10, and this is not specifically limited in the embodiments of the present application. Accordingly, when the image acquisition device 20 and the point cloud acquisition device 30 are modules with corresponding functions integrated on the terminal 10, the terminal 10 may obtain the image acquired by the image acquisition device 20 and the point cloud acquired by the point cloud acquisition device 30 by means of internal data transmission.
[0071] Please refer to Figure 2 , which shows a schematic diagram of a process of a point cloud coloring method provided by the present application. As an example but not a limitation, the method is applied in the above-mentioned terminal.
[0072] S201, the terminal obtains multiple original point clouds and multiple images.
[0073] The image is an image captured by an image acquisition device. The original point cloud is a point cloud captured by a point cloud acquisition device. In some embodiments, the original point cloud is an unprocessed point cloud. In some embodiments, the original point cloud is a point cloud obtained by stitching together multiple frames of point clouds. Accordingly, the terminal acquires multiple frames of point clouds, maps the multiple frames of point clouds into the same coordinate system, and obtains a stitched original point cloud. In this implementation, the original point cloud is obtained by stitching the point clouds into the same coordinate system, so that the data points in the original point cloud are denser, thereby improving the recognizability of the point cloud.
[0074] In some embodiments, the terminal receives the original point cloud sent by the point cloud acquisition device and the image sent by the image acquisition device in real time. In some embodiments, the terminal periodically receives the original point cloud sent by the point cloud acquisition device and the image sent by the image acquisition device. Among them, the period at which the terminal receives the original point cloud and the image is the same or different, and the period at which the terminal receives the original point cloud and the image can be set as needed. In the embodiments of the present application, there is no specific limitation on this. In some embodiments, the terminal obtains multiple original point clouds and multiple images from a database. Accordingly, the image acquisition device uploads the acquired images to the database, the point cloud acquisition device uploads the acquired point cloud to the database, and the terminal obtains the original point cloud and the image from the database.
[0075] S202: For any original point cloud, the terminal determines, from a plurality of images, a target image that matches the original point cloud based on a first generation time of the original point cloud.
[0076] Among them, for the original point cloud composed of a single frame point cloud, the first generation time is the time when the point cloud acquisition device acquires the original point cloud. For the original point cloud composed of multiple frame point clouds, the first generation time can be the average generation time of the multiple frame point clouds, or the first generation time can be the generation time of the earliest generated point cloud in the multiple frame point clouds. In the embodiment of the present application, there is no specific limitation on this.
[0077] Since the original point cloud is collected by the point cloud collection device, and the image is collected by the image collection device, the collection frequencies of the two are usually different. Therefore, it is necessary to synchronize the original point cloud with the image, that is, to determine the target image that matches the original point cloud. In this step, the terminal determines the target image that matches the original point cloud through the first generation time of the original point cloud and the second generation time of the image.
[0078] Accordingly, this step includes: the terminal determines the second generation time of each image; and from the multiple images, determines a target image whose second generation time matches the first generation time. The original point cloud matches the target image when the second generation time is the same as the first generation time, or when the time difference between the second generation time and the first generation time is the smallest.
[0079] In an embodiment of the present application, a target image matching the original point cloud is determined by the first generation time of the original point cloud and the second generation time of the image, so that the time difference between the original point cloud and the target image is minimized, thereby preventing coloring deviations caused by the original point cloud and the target image being generated at different times, and improving the accuracy of coloring.
[0080] S203, the terminal projects the original point cloud onto the target image that matches the original point cloud, and obtains the correspondence between the data points of the original point cloud and the pixel points of the target image.
[0081] The correspondence between the data points of the original point cloud and the pixel points of the target image refers to the correspondence between the data points of the original point cloud and the pixel points of the target image in the same coordinate system.
[0082] In some embodiments, since the point cloud acquisition device is installed on the vehicle, the point cloud is collected during the driving process of the vehicle, and a frame of point cloud needs to be collected for a period of time, generally 100 milliseconds, so the collected point cloud may be distorted. Therefore, the terminal performs a dedistortion process on the original point cloud before projecting it onto the matching target image. The dedistortion method includes: the terminal determines the moving speed and direction of the point cloud acquisition device that collects the original point cloud; and performs a dedistortion process on the original point cloud based on the moving speed and direction.
[0083] The terminal performs dedistortion processing on the original point cloud according to the moving speed and direction of the point cloud acquisition device through pure estimation method, sensor-assisted method or fusion method. Through this solution, the coloring deviation caused by the deviation in the correspondence between the pixel points of the image and the data points of the point cloud caused by the point cloud distortion is avoided, and the accuracy of coloring is improved.
[0084] S204: The terminal colors the original point cloud based on the correspondence between the data point and the pixel point to obtain a colored point cloud.
[0085] After the terminal determines the correspondence between the data point and the pixel point, each data point corresponds to a pixel coordinate, and the data point is colored according to the color value of the pixel point corresponding to the pixel coordinate. This process can be implemented through the following steps S2041-S2043, including:
[0086] S2041: For any data point in the original point cloud, the terminal determines a pixel point that matches the coordinates of the data point based on the correspondence between the data point and the pixel point.
[0087] The correspondence between the data point and the pixel point represents the data point and the pixel point at the same coordinate position in the same coordinate system. Therefore, in this step, the terminal determines the pixel point at the same position as the data point based on the position of the data point in the coordinate system.
[0088] S2042, the terminal assigns the color value of the pixel to the matching data point to obtain a colored data point that matches the data point.
[0089] The color value may be any value that can represent the color of a pixel, for example, the color value is an RGB value. In this step, the terminal assigns the color value of the pixel to the data point so that the data point has a color value, thereby achieving the purpose of coloring the data point.
[0090] S2043, the terminal combines the colored data points that match the multiple data points of the original point cloud into the colored point cloud.
[0091] The terminal traverses multiple data points in the original point cloud, assigns a value to each data point, and composes the assigned data points into a colored point cloud.
[0092] In the embodiment of the present application, since the data points and pixel points are projected into the same coordinate system, the data points and pixel points with matching coordinate positions are formed into a corresponding relationship, which reduces the position difference between the data points and pixel points in the real scene, thereby ensuring the accuracy of data point coloring.
[0093] The colored point cloud is used to generate sample data for application in model training scenarios. For example, in the field of autonomous driving, lane lines, road signs, and obstacles are identified through perception models. The perception model can be trained using sample data composed of point clouds. Accordingly, the terminal determines sample data based on the colored point cloud and performs model training based on the sample data.
[0094] In some embodiments, the colored point cloud is annotated by manual annotation to obtain sample data. Accordingly, the terminal displays the colored point cloud, is used to display the colored point cloud to the annotator, receives annotation information of the colored point cloud, annotates the colored point cloud based on the annotation information, obtains sample data, and performs model training based on the sample data.
[0095] In an embodiment of the present application, the data points in the original point cloud are colored by the correspondence between the data points in the original point cloud and the pixel points of the image. This scheme makes up for the lack of color information in the point cloud, making the target object represented in the point cloud clearer, so that the target object in the point cloud is easy to identify, thereby improving the efficiency of manual annotation of the point cloud.
[0096] Please refer to Figure 3 , which shows a schematic diagram of a process of a point cloud coloring method provided by the present application. As an example but not a limitation, the method is applied in the above-mentioned terminal.
[0097] S301, the terminal obtains multiple original point clouds and multiple images.
[0098] The principle of this step is the same as that of step S201 and will not be described again here.
[0099] S302: For any original point cloud, the terminal determines, from a plurality of images, a target image that matches the original point cloud based on a first generation time of the original point cloud.
[0100] The principle of this step is the same as that of step S202 and will not be described again here.
[0101] S303: The terminal maps the data points of the original point cloud from the first coordinate system to the second coordinate system.
[0102] The first coordinate system is a three-dimensional coordinate system corresponding to a point cloud acquisition device that acquires the original point cloud, and the second coordinate system is a three-dimensional coordinate system corresponding to an image acquisition device that acquires the target image.
[0103] The terminal calibrates the point cloud acquisition device and the image acquisition device through the device parameters of the point cloud acquisition device and the device parameters of the image acquisition device, obtains the first coordinate system and the second coordinate system, determines the positional relationship between the first coordinate system and the world coordinate system, and determines the positional relationship between the first coordinate system and the world coordinate system, thereby determining the positional relationship between the first coordinate system and the second coordinate system according to the positional relationship between the first coordinate system and the second coordinate system and the world coordinate system, and based on the positional relationship between the first coordinate system and the second coordinate system, performs matrix transformation on the first coordinate system to obtain the position coordinates of the point cloud in the second coordinate system. Among them, the terminal can complete the transformation of the coordinate system through matrix multiplication and addition.
[0104] S304, based on the device parameters of the image acquisition device, the terminal projects the original point cloud in the second coordinate system to the plane where the target image matched by the original point cloud is located, and obtains the correspondence between the data points of the original point cloud and the pixel points of the target image.
[0105] The device parameters of the image acquisition device refer to the parameters used by the image acquisition device when acquiring images. For example, when the image acquisition device is a camera, the device parameters include the intrinsic parameters and extrinsic parameters of the camera.
[0106] In some embodiments, the plane where the target image is located is the image coordinate system corresponding to the target image. In this step, the terminal projects the original point cloud in the second coordinate system to the plane where the matching target image is located, which means: the terminal keeps the X coordinate and Y coordinate of the data point in the original point cloud unchanged, sets the Z coordinate of the data point to zero, that is, projects the data point in the original point cloud to the XY plane of the second coordinate system, and then, according to the coordinate system of the plane where the target image is located, aligns the XY plane of the second coordinate system with the plane where the target image is located. For example, align the origin of the XY plane of the second coordinate system with the center point of the target image, or align the origin of the XY plane of the second coordinate system with the upper left corner of the target image. The terminal determines the two points closest to the data point on the XY plane and the pixel point of the image in the same coordinate system as a set of data points and pixel points with a corresponding relationship.
[0107] In some embodiments, the terminal projects all data points in the original point cloud onto the plane where the target image matching the original point cloud is located. In some embodiments, the terminal deletes some data points in the original point cloud and projects other data points onto the plane where the target image is located. This process can be implemented by the following steps S3041-S3043, including:
[0108] S3041: The terminal determines the viewing cone of the image acquisition device based on the device parameters.
[0109] The visual cone refers to the visible range of the image acquisition device when acquiring images, and is usually composed of four planes. Based on the device parameters of the image acquisition device, the terminal determines the positions of the four planes corresponding to the visual cone of the image acquisition device in the second coordinate system in the second coordinate system. The position of the plane can be the plane equation of the plane in the second coordinate system.
[0110] S3042: The terminal deletes the data points of the original point cloud that are outside the viewing cone in the second coordinate system to obtain a valid point cloud.
[0111] This step can be achieved through the following steps, including: the terminal determines the positions of multiple planes of the viewing cone in the second coordinate system; based on the coordinates of the data points in the original point cloud in the second coordinate system and the positions of the multiple planes in the second coordinate system, deletes the data points outside the range of the multiple planes of the viewing cone to obtain the valid point cloud.
[0112] S3043: The terminal projects the data points of the valid point cloud in the second coordinate system to the plane where the target image matched by the original point cloud is located, and obtains the correspondence between the data points of the original point cloud and the pixel points of the target image.
[0113] The terminal projects the data points of the valid point cloud onto the XY plane of the second coordinate system, and aligns the image with the projection points to obtain the corresponding relationship between the pixel points and the data points.
[0114] In an embodiment of the present application, when determining the correspondence between the data points of the original point cloud and the pixel points of the target image, the data points outside the viewing cone in the original point cloud data are deleted, thereby preventing the data points outside the viewing cone from being colored and improving the accuracy of point cloud coloring.
[0115] S305: The terminal colors the original point cloud based on the correspondence between the data point and the pixel point to obtain a colored point cloud.
[0116] The principle of this step is the same as that of step S204 and will not be described again.
[0117] In an embodiment of the present application, the data points in the original point cloud are colored by the correspondence between the data points in the original point cloud and the pixel points of the image. This scheme makes up for the lack of color information in the point cloud, making the target object represented in the point cloud clearer, so that the target object in the point cloud is easy to identify, thereby improving the efficiency of manual annotation of the point cloud.
[0118] It should be understood that the order of execution of the steps in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above embodiments can be combined with multiple embodiments to implement the method of point cloud coloring. Figure 4 , which shows a schematic diagram of a process of a point cloud coloring method provided by the present application. As an example but not a limitation, the method is applied in the above-mentioned terminal.
[0119] S401, the terminal obtains multiple original point clouds and multiple images.
[0120] S402: The terminal matches the original point cloud with the image based on the first time and the second time to obtain a target image matched with the original point cloud.
[0121] S403: The terminal performs dedistortion processing on the original point cloud.
[0122] S404: The terminal maps the original point cloud after the dedistortion processing to a second coordinate system corresponding to the target image.
[0123] S405: The terminal determines the viewing cone of the image acquisition device.
[0124] S406, the terminal deletes the data points outside the viewing cone in the original point cloud after distortion to obtain a valid point cloud.
[0125] S407, the terminal projects the data points in the valid point cloud onto the plane where the target image is located.
[0126] S408, the terminal determines the correspondence between the data points and the pixel points according to the positional relationship between the data points and the pixel points on the plane where the target image is located.
[0127] S409: The terminal colors the original point cloud based on the correspondence between the data point and the pixel point to obtain a colored point cloud.
[0128] In an embodiment of the present application, the data points in the original point cloud are colored by the correspondence between the data points in the original point cloud and the pixel points of the image. This scheme makes up for the lack of color information in the point cloud, making the target object represented in the point cloud clearer, so that the target object in the point cloud is easy to identify, thereby improving the efficiency of manual annotation of the point cloud.
[0129] See also Figure 5 , which shows a schematic diagram of the structure of a point cloud coloring device provided by the present application, including units for executing the various steps in the above embodiments, see Figure 5 , the point cloud coloring device comprises:
[0130] An acquisition unit 501 is used to acquire multiple original point clouds and multiple images;
[0131] A first determining unit 502 is configured to determine, for any original point cloud, a target image matching the original point cloud from a plurality of images based on a first generation time of the original point cloud;
[0132] A projection unit 503 is used to project the original point cloud onto the target image that matches the original point cloud to obtain a correspondence between data points of the original point cloud and pixel points of the target image;
[0133] The coloring unit 504 is used to color the original point cloud based on the corresponding relationship between the data point and the pixel point to obtain a colored point cloud.
[0134] In some embodiments, the projection unit 503 is used to map the data points of the original point cloud from a first coordinate system to a second coordinate system, where the first coordinate system is a three-dimensional coordinate system corresponding to a point cloud acquisition device that acquires the original point cloud, and the second coordinate system is a three-dimensional coordinate system corresponding to an image acquisition device that acquires the target image; based on the device parameters of the image acquisition device, the original point cloud in the second coordinate system is projected to the plane where the target image matching the original point cloud is located, to obtain the correspondence between the data points of the original point cloud and the pixel points of the target image.
[0135] In some embodiments, the projection unit 503 is used to determine the viewing cone of the image acquisition device based on the device parameters; delete the data points of the original point cloud outside the viewing cone in the second coordinate system to obtain a valid point cloud; project the data points of the valid point cloud in the second coordinate system to the plane where the target image matched by the original point cloud is located, to obtain the correspondence between the data points of the original point cloud and the pixel points of the target image.
[0136] In some embodiments, the projection unit 503 is used to determine the positions of multiple planes of the viewing cone in the second coordinate system; based on the coordinates of the data points in the original point cloud in the second coordinate system and the positions of the multiple planes in the second coordinate system, the data points outside the range of the multiple planes of the viewing cone are deleted to obtain the valid point cloud.
[0137] In some embodiments, the shading unit 504 is used to determine, for any data point in the original point cloud, a pixel point that matches the coordinates of the data point based on the correspondence between the data point and the pixel point; assign the color value of the pixel point to the matched data point to obtain a colored data point that matches the data point; and form the colored point cloud from colored data points that match multiple data points of the original point cloud.
[0138] In some embodiments, the apparatus further comprises:
[0139] A first determining unit 502 is used to determine the moving speed and direction of the point cloud acquisition device that acquires the original point cloud;
[0140] The dedistortion unit is used to perform dedistortion processing on the original point cloud based on the moving speed and direction.
[0141] In some embodiments, the first determining unit 502 is used to determine the second generation time of each of the images; and to determine, from the plurality of images, a target image whose second generation time matches the first generation time.
[0142] In some embodiments, the apparatus further comprises:
[0143] A third determining unit, configured to determine sample data based on the colored point cloud;
[0144] The model training unit is used to perform model training based on the sample data.
[0145] In an embodiment of the present application, the data points in the original point cloud are colored by the correspondence between the data points in the original point cloud and the pixel points of the image. This scheme makes up for the lack of color information in the point cloud, making the target object represented in the point cloud clearer, so that the target object in the point cloud is easy to identify, thereby improving the efficiency of manual annotation of the point cloud.
[0146] Figure 6 Schematic diagram of a terminal device provided by an embodiment of the present application. Figure 6 As shown, the terminal device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60, such as a point cloud coloring program. When the processor 60 executes the computer program 62, the steps in the above-mentioned point cloud coloring method embodiments are implemented, such as Figure 2 Alternatively, when the processor 60 executes the computer program 62, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of units 501 to 504 are shown.
[0147] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 62 in the point cloud coloring terminal device 6. For example, the computer program 62 may be divided into a synchronization module, a summary module, an acquisition module, and a return module (module in a virtual device), and the specific functions of each module are as follows:
[0148] An acquisition unit 501 is used to acquire multiple original point clouds and multiple images;
[0149] A first determining unit 502 is configured to determine, for any original point cloud, a target image matching the original point cloud from a plurality of images based on a first generation time of the original point cloud;
[0150] A projection unit 503 is used to project the original point cloud onto the target image that matches the original point cloud to obtain a correspondence between data points of the original point cloud and pixel points of the target image;
[0151] The coloring unit 504 is used to color the original point cloud based on the corresponding relationship between the data point and the pixel point to obtain a colored point cloud.
[0152] The terminal device 6 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 6 It is only an example of the terminal device 6 and does not constitute a limitation of the terminal device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.
[0153] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0154] The memory 61 may be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. The memory 61 may also be an external storage device of the terminal device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the terminal device 6. The memory 61 is used to store the computer program and other programs and data required by the terminal device. The memory 61 may also be used to temporarily store data that has been output or is to be output.
[0155] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0156] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0157] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0158] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0160] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0161] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0162] An embodiment of the present application further provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.
[0163] The embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0164] The embodiment of the present application also provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0165] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A point cloud coloring method, characterized in that: The method comprises: Get multiple raw point clouds and multiple images; For any original point cloud, based on the first generation time of the original point cloud, determining a target image matching the original point cloud from a plurality of images; Projecting the original point cloud onto the target image that matches the original point cloud to obtain a correspondence between data points of the original point cloud and pixel points of the target image; Based on the correspondence between the data points and the pixel points, the original point cloud is colored to obtain a colored point cloud.
2. The method according to claim 1, characterized in that The projecting the original point cloud onto the target image matching the original point cloud to obtain a correspondence between data points of the original point cloud and pixel points of the target image includes: Mapping the data points of the original point cloud from a first coordinate system to a second coordinate system, wherein the first coordinate system is a three-dimensional coordinate system corresponding to a point cloud acquisition device that acquires the original point cloud, and the second coordinate system is a three-dimensional coordinate system corresponding to an image acquisition device that acquires the target image; Based on the device parameters of the image acquisition device, the original point cloud in the second coordinate system is projected to the plane where the target image matched by the original point cloud is located, so as to obtain the correspondence between the data points of the original point cloud and the pixel points of the target image.
3. The method according to claim 2, characterized in that The method of projecting the original point cloud in the second coordinate system onto a plane where a target image matched with the original point cloud is located based on the device parameters of the image acquisition device, and obtaining a correspondence between data points of the original point cloud and pixel points of the target image, comprises: determining a viewing cone of the image acquisition device based on the device parameters; Deleting data points of the original point cloud outside the viewing cone in the second coordinate system to obtain a valid point cloud; The data points of the valid point cloud in the second coordinate system are projected onto the plane where the target image matched by the original point cloud is located, so as to obtain the corresponding relationship between the data points of the original point cloud and the pixel points of the target image.
4. The method according to claim 3, characterized in that The step of deleting data points of the original point cloud outside the viewing cone in the second coordinate system to obtain a valid point cloud includes: Determining positions of a plurality of planes of the viewing cone in the second coordinate system; Based on the coordinates of the data points in the original point cloud in the second coordinate system and the positions of the multiple planes in the second coordinate system, the data points outside the range of the multiple planes of the viewing cone are deleted to obtain the valid point cloud.
5. The method according to claim 1, characterized in that The coloring of the original point cloud based on the correspondence between the data points and the pixel points to obtain a colored point cloud includes: For any data point in the original point cloud, based on the correspondence between the data point and the pixel point, determine a pixel point that matches the coordinates of the data point; Assigning the color value of the pixel point to the matched data point to obtain a colored data point matched by the data point; The colored point cloud is composed of colored data points that match multiple data points of the original point cloud.
6. The method according to claim 1, characterized in that Before projecting the original point cloud onto the target image matching the original point cloud to obtain the corresponding relationship between the data points of the original point cloud and the pixel points of the target image, the method further includes: Determining a moving speed and direction of a point cloud acquisition device that acquires the original point cloud; The original point cloud is dedistorted based on the moving speed and direction.
7. The method according to claim 1, characterized in that The step of determining, based on the first generation time of the original point cloud, a target image to be matched by the original point cloud from the plurality of images comprises: determining a second generation time of each of the images; A target image whose second generation time matches the first generation time is determined from the plurality of images.
8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Determining sample data based on the colored point cloud; Model training is performed based on the sample data.
9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to implement the point cloud coloring method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the point cloud coloring method according to any one of claims 1 to 8.