Point cloud data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a point cloud data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: generating two-dimensional images respectively corresponding to a target scene at a plurality of visual angles; for each two-dimensional image, determining a first semantic tag of each pixel point in the two-dimensional image; for each data point in the three-dimensional point cloud of the target scene, detecting the visibility of the data point at a plurality of visual angles to obtain a detection result; for each data point, if the detection result represents that the data point is visible at a target view angle in the plurality of view angles, determining a target two-dimensional image corresponding to the target view angle from the plurality of two-dimensional images, and determining a first pixel point corresponding to a projection point of the data point at the target view angle from each pixel point of the target two-dimensional image, taking the first semantic tag of the first pixel point as a target first semantic tag of the data point; and determining a second semantic tag of the data point based on the target first semantic tag of the data point. According to the invention, the accuracy of semantic segmentation of the point cloud can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device and storage medium for processing point cloud data. Background Technology

[0002] Computer vision is a core branch of artificial intelligence, aiming to enable machines to "understand" image content. Semantic annotation of 3D point clouds is a core task in scene understanding within computer vision. Currently, 3D point cloud semantic annotation is achieved through single-view semantic segmentation, which suffers from inaccurate segmentation and difficulty in guaranteeing semantic precision due to occlusion and projection errors caused by the single viewpoint. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for processing point cloud data, which can improve the accuracy of point cloud semantic segmentation.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a method for processing point cloud data, including: Generate two-dimensional images of the target scene from multiple perspectives; For each of the two-dimensional images, determine the first semantic label for each pixel in the two-dimensional image; For each data point in the 3D point cloud of the target scene, the visibility of the data points in multiple viewpoints is detected to obtain the detection results; For each of the data points, if the detection result indicates that the data point is visible in the target view of the multiple viewpoints, the target two-dimensional image corresponding to the target viewpoint is determined from the multiple two-dimensional images, and the first pixel point corresponding to the projection point of the data point in the target viewpoint is determined from each pixel point of the target two-dimensional image, and the first semantic label of the first pixel point is used as the target first semantic label of the data point. For each data point, a second semantic label is determined based on the target first semantic label of the data point.

[0005] This application embodiment also provides a point cloud data processing apparatus, including: The generation module is used to generate two-dimensional images of the target scene from multiple viewpoints. The first determining module is used to determine the first semantic label of each pixel in the two-dimensional image for each of the two-dimensional images; The detection module is used to detect the visibility of each data point in the 3D point cloud of the target scene from multiple viewpoints and obtain the detection results. The second determining module is used for each of the data points. If the detection result indicates that the data point is visible in the target view of the multiple viewpoints, it determines the target two-dimensional image corresponding to the target viewpoint from the multiple two-dimensional images, determines the first pixel point corresponding to the projection point of the data point in the target viewpoint from each pixel point of the target two-dimensional image, and uses the first semantic label of the first pixel point as the target first semantic label of the data point. The third determining module is used to determine the second semantic label of each data point based on the target first semantic label of the data point.

[0006] In the above scheme, the generation module is further configured to construct a three-dimensional scene model of the target scene; determine the camera pose of the virtual camera with the viewpoint for each viewpoint; and project the three-dimensional scene model onto the image plane of the virtual camera of the viewpoint based on the camera pose of the viewpoint for each viewpoint to obtain the two-dimensional image corresponding to the viewpoint.

[0007] In the above scheme, the generation module is further configured to acquire two-dimensional scene images of the target scene captured from a first perspective, and acquire the camera pose of the camera with the first perspective; initialize based on the two-dimensional scene images to obtain an initial three-dimensional scene model; project the initial three-dimensional representation onto the image plane of the camera based on the camera pose to obtain a predicted two-dimensional image; update the initial three-dimensional scene model based on the difference between the predicted two-dimensional image and the two-dimensional scene image to obtain the three-dimensional scene model.

[0008] In the above scheme, the first determining module is further configured to obtain the descriptive text of the two-dimensional image, the descriptive text being used to describe the target object in the two-dimensional image; encode the two-dimensional image to obtain image features, and encode the descriptive text to obtain text features; fuse the image features and the text features to obtain fused features; and based on the fused features, predict the first semantic label of each pixel using a machine learning model.

[0009] In the above scheme, the detection module is further configured to perform the following processing for each of the viewpoints: determine the camera center point of the virtual camera with the viewpoint, and determine the distance between the data point and the camera center point; obtain the depth value of the pixel corresponding to the data point in the two-dimensional image of the viewpoint, and determine the absolute value of the difference between the distance and the depth value; if the absolute value is less than a preset threshold, determine a first detection result, the first detection result indicating that the data point is visible in the viewpoint; if the absolute value is greater than or equal to the preset threshold, determine a second detection result, the second detection result indicating that the data point is not visible in the viewpoint, the detection result including the first detection result and the second detection result.

[0010] In the above scheme, the second determining module is further configured to determine the target camera pose of the target virtual camera having the target viewpoint before determining the first pixel point corresponding to the projection point of the data point in the target viewpoint from each pixel point of the target two-dimensional image; based on the target camera pose, project the data point onto the image plane of the target virtual camera to obtain the projection point of the data point in the target viewpoint.

[0011] In the above scheme, the third determining module is further configured to: if the number of target viewpoints is one, determine the target first semantic tag as the second semantic tag; if the number of target viewpoints is multiple and the target first semantic tags of the multiple target viewpoints are unique, determine the target first semantic tag as the second semantic tag; if the number of target viewpoints is multiple and the target first semantic tags of the multiple target viewpoints are not unique, for each target first semantic tag, obtain the number of target viewpoints corresponding to the target first semantic tag, and determine the target first semantic tag corresponding to the largest number as the second semantic tag.

[0012] This application also provides an electronic device, including: Memory is used to store executable instructions for a computer; The processor, when executing computer-executable instructions stored in the memory, implements the point cloud data processing method provided in the embodiments of this application.

[0013] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the point cloud data processing method provided in this application.

[0014] This application also provides a computer program product, including computer-executable instructions or a computer program, which, when executed by a processor, implements the point cloud data processing method provided in this application.

[0015] The embodiments of this application have the following beneficial effects: by generating multi-view 2D images of the target scene, occluded areas that cannot be observed from a single viewpoint are covered; a first semantic label is determined for each pixel in the 2D image from each viewpoint, providing a basis for multi-view point cloud semantic determination; the visibility of data points in the 3D point cloud is detected from multiple views, and visible target views are selected to avoid interference from incorrect semantic labels from invisible views; then, a second semantic label is determined based on the first semantic label of the data points in the visible target views, realizing multi-view point cloud semantic determination. Thus, the problem of semantic segmentation errors caused by single-view occlusion is effectively solved, improving the accuracy of 3D point cloud semantic segmentation. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the architecture of the point cloud data processing system provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 3 This is a first flowchart illustrating the point cloud data processing method provided in this application embodiment; Figure 4 This is a schematic diagram of the second process of the point cloud data processing method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the third process of the point cloud data processing method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the fourth process of the point cloud data processing method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the fifth process of the point cloud data processing method provided in the embodiments of this application.

[0017] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0020] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0021] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of a larger module or unit that includes the functionality of the module or unit.

[0022] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for descriptive purposes only and is not intended to limit the scope of this application.

[0023] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0024] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0025] 1) A 3D point cloud is a set of discrete points in a 3D space representing a target scene. Each point contains 3D coordinates (...). The point cloud, along with its possible attributes such as color and normal vectors, is used to accurately represent the geometric structure of a scene. It is typically generated by LiDAR, multi-view stereo, or 3D reconstruction technologies and is the core object of 3D point cloud semantic annotation.

[0026] 2) Viewpoint is a comprehensive description of the angle and direction at which the camera observes the target scene, and it is uniquely determined by the camera pose (position and orientation). It answers the questions of "from where to look" (camera position) and "to which direction to look" (camera orientation), and is the basis for generating multi-view 2D images. By overlaying multiple viewpoints, the occlusion problem of a single viewpoint can be solved.

[0027] 3) Semantic labels are semantic category annotations for pixels in a 2D image or data points in a 3D point cloud (such as "pedestrian" or "wall"), which associate low-level visual data with high-level semantic understanding. In this application, the first semantic label corresponds to the semantic category of pixels in a 2D image (for example, it can be obtained from a text-image segmentation model), and the second semantic label corresponds to the final semantic category of data points in a 3D point cloud (the result of fusing multiple visible perspectives).

[0028] 4) Visibility is the attribute of whether data points in a 3D point cloud can be observed by a virtual camera from a specific viewpoint. The judgment logic includes "whether the point is within the camera's view frustum" and "whether it is not occluded by other objects". It is a key filtering condition for multi-view semantic fusion, which avoids erroneous interference from invisible viewpoints by retaining only the first semantic label of the visible target viewpoint.

[0029] 5) 3D Gaussian Splatting (3D GS) is an explicit 3D scene representation technique that fits the scene geometry and appearance using a large number of 3D Gaussian distributions (each Gaussian containing position, shape, color, and opacity). It supports efficient differentiable rendering and can quickly generate multi-view 2D images (i.e., RGB images) and depth maps, covering single-view occlusion areas. It is the representation method for the target scene in this application; through 3D Gaussian sputtering, a 3D scene model of the target scene is constructed.

[0030] 6) Camera pose is a set of parameters describing the camera's state in the world coordinate system, including extrinsic parameters (rotation matrix). With translation vector (determines position and orientation) and internal parameters (focal length) Main point (Etc., determining the projection method). Extrinsic parameters define "where the camera is and which way it's looking," while intrinsic parameters define "how the scene is projected onto the image," forming the basis for generating the viewpoint and 2D image.

[0031] 7) The image plane of the virtual camera is the plane on which the virtual camera projects a 3D scene into a 2D image (perpendicular to the camera's optical axis). 3D points are projected onto this image plane through intrinsic parameters to form 2D pixel coordinates. It is the physical carrier for generating multi-view 2D images, ensuring a one-to-one correspondence between scene geometry and pixels in the 2D image.

[0032] 8) The camera center point is the geometric center of the camera in the world coordinate system, corresponding to the translation vector (camera position) in the extrinsic parameters. It is the "perspective center" of the 3D points projected onto the image plane (all projection lines pass through this point), and is the core reference point for calculating the projection position (projection point) of data points in the 3D point cloud.

[0033] 9) Depth is the axial distance from a 3D point to the center point of the camera, usually referring to the distance in the camera coordinate system. Axis coordinates ( (Axis points towards the scene). The larger the depth value, the farther the point is from the camera; conversely, the smaller the value, the closer it is. A depth map is a set of depth values ​​for each pixel in a two-dimensional image, and it is an important geometric basis for determining the visibility and semantic fusion of data points in a point cloud.

[0034] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for processing point cloud data, which can improve the accuracy of point cloud semantic segmentation. The embodiments of this application will now be described in detail based on the above description of the terms and concepts used in the embodiments.

[0035] The following describes the point cloud data processing system provided in the embodiments of this application. See also... Figure 1 , Figure 1 This is a schematic diagram of the architecture of a point cloud data processing system provided in an embodiment of this application. To support an exemplary application, the point cloud data processing system 100 includes: a server 200, a network 300, and a terminal 400. The terminal 400 is connected to the server 200 via the network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both, using wireless or wired links to achieve data transmission.

[0036] Here, in response to the point cloud semantic determination instruction for the 3D point cloud, the terminal 400 sends a point cloud semantic determination request for the 3D point cloud to the server 200; the server 200 receives the point cloud semantic determination request sent by the terminal 400; in response to the point cloud semantic determination request, it generates two-dimensional images of the target scene corresponding to multiple viewpoints respectively; for each two-dimensional image, it determines the first semantic label of each pixel in the two-dimensional image; for each data point in the 3D point cloud of the target scene, it detects the visibility of the data point in multiple viewpoints and obtains the detection result; for each data point, if the detection result indicates that the data point is visible in the target viewpoint in multiple viewpoints, it determines the target two-dimensional image corresponding to the target viewpoint from the multiple two-dimensional images, and determines the first pixel corresponding to the projection point of the data point in the target viewpoint from each pixel of the target two-dimensional image, and uses the first semantic label of the first pixel as the target first semantic label of the data point; for each data point, it determines the second semantic label of the data point based on the target first semantic label of the data point; it sends the second semantic label of each data point in the 3D point cloud to the terminal 400; the terminal 400 receives the second semantic label of each data point in the 3D point cloud sent by the server 200.

[0037] The point cloud data processing method provided in this application embodiment is implemented by an electronic device. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server working together. The electronic device implementing the point cloud data processing method provided in this application embodiment can be various types of terminals or servers. The server (e.g., server 200) can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal (e.g., terminal 400) can be a laptop, tablet, desktop computer, smartphone, smart voice interaction device (e.g., smart speaker), smart home appliance (e.g., smart TV), smartwatch, vehicle terminal, wearable device, virtual reality (VR) device, aircraft, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, and this application embodiment does not impose any restrictions on this.

[0038] In some embodiments, the terminal or server can implement the point cloud data processing method provided in this application embodiment by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0039] The following describes an electronic device that implements a point cloud data processing method according to embodiments of this application. See also... Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 500 provided in this embodiment can be a terminal or a server. Figure 2 As shown, electronic device 500 includes at least one processor 510, memory 550, at least one network interface 520, and user interface 530. The various components in electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 540.

[0040] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0041] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0042] Memory 550 may be removable, non-removable, or a combination thereof. Memory 550 may include one or more storage devices physically located away from processor 510. Memory 550 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0043] In some embodiments, memory 550 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below. Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks; network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including Bluetooth, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB); presentation module 553 is used to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 associated with user interface 530 (e.g., a display screen, a speaker, etc.); input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0044] In some embodiments, the point cloud data processing apparatus provided in this application can be implemented in software. Figure 2 A processing device 555 for point cloud data stored in memory 550 is shown. It can be software in the form of programs and plug-ins, including the following software modules: generation module 5551, first determination module 5552, detection module 5553, second determination module 5554 and third determination module 5555. These modules are logical and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0045] The following describes the point cloud data processing method provided in the embodiments of this application. As mentioned above, the point cloud data processing method provided in the embodiments of this application is implemented by an electronic device, such as a server or terminal alone, or a server and terminal working together. Therefore, the executing entity of each step will not be described again below. See Figure 3 , Figure 3 This is a first flowchart illustrating the point cloud data processing method provided in this application embodiment. The point cloud data processing method provided in this application embodiment includes: Step 101: Generate two-dimensional images of the target scene from multiple viewpoints.

[0046] For step 101, a 3D scene model of the target scene can be constructed first, such as a 3D scene model based on 3D Gaussian sputtering. Then, based on this 3D scene model, 2D images of each viewpoint are obtained through rendering from multiple perspectives. Here, a 3D Gaussian sputtering model (i.e., a 3D scene model) is trained using a set of 2D images (i.e., RGB images) taken from multiple different first-person perspectives and their corresponding camera poses. This 3D Gaussian sputtering model explicitly represents the target scene as a set of a large number of 3D Gaussian distributions. Each Gaussian distribution includes parameters such as center position (mean), covariance matrix (determined by scaling factor and rotation quaternion), opacity, and color information (represented by spherical harmonic function). At this point, the target scene has been encoded as an explicit 3D structure that can be accurately rendered from any perspective, providing the underlying foundation for the subsequent generation of 2D images from multiple perspectives. The "multiple perspectives" are not actual physical perspectives, but rather a set of camera poses from artificially set virtual cameras. In practical applications, camera pose design follows a "covering occluded areas" strategy, including: a) distributing camera poses around the target scene (e.g., evenly distributed at scene boundaries, filling blind spots in the original viewpoint (i.e., the shooting viewpoint of the 3D point cloud to be semantically annotated); b) focusing on covering areas in the original viewpoint that cannot be observed due to object occlusion (e.g., the back of objects, corners, under furniture, etc.); c) ultimately achieving the goal of "observing the scene from more dimensions," overcoming the viewpoint limitations of 3D point clouds. Utilizing the fast differentiable rendering capability of 3D Gaussian sputtering (based on the differentiable calculation of ray projection and Gaussian distribution), the following operations are performed on the camera pose of each virtual camera: a) calculating the interaction (color, depth, etc.) between the light rays and all 3D Gaussians in the 3D scene model under that viewpoint (i.e., camera pose); b) outputting a high-quality, highly realistic 2D RGB image corresponding to that viewpoint. (Textures, lighting details are consistent with the original scene) and depth map .

[0047] In some embodiments, see Figure 4Step 101, “Generate two-dimensional images of the target scene corresponding to multiple viewpoints”, can be achieved by executing the following steps 1011-1013: Step 1011, construct a three-dimensional scene model of the target scene; Step 1012, determine the camera pose of the virtual camera with the viewpoint for each viewpoint; Step 1013, for each viewpoint, based on the camera pose of the viewpoint, project the three-dimensional scene model onto the image plane of the virtual camera of the viewpoint to obtain the two-dimensional image corresponding to the viewpoint.

[0048] For step 1011, obtain a set of two-dimensional RGB images of the target scene taken from multiple different original viewpoints (i.e., first-person viewpoints). and a set of camera poses for cameras that capture 2D RGB images. ,in, It includes camera extrinsic parameters (rotation matrix and translation vector) and intrinsic parameters (focal length and principal point coordinates). Based on 3D Gaussian sputtering (3D GS) technology, an initial 3D scene model of the target scene is initialized, i.e. A set of three-dimensional Gaussian distributions Each Gaussian From parameter set Define, where, The coordinates are in the world coordinate system centered at Gauss. The covariance matrix is ​​calculated from the scaling factor and the rotation quaternion. The opacity is Gaussian. The color information is Gaussian. The rendering loss function is minimized using an optimizer (such as the Adam optimizer). Iteratively update all Gaussian parameters until convergence to obtain a 3D scene model.

[0049] For step 1012, "multiple perspectives" do not refer to the actual physical perspectives of the shots, but rather to a collection of camera poses from a virtual camera that is artificially set. Each camera pose includes the virtual camera's extrinsic parameters (rotation matrix and translation vector) and intrinsic parameters (focal length and principal point coordinates). The camera pose design follows a "covering occluded areas" strategy, including: a) the camera pose distribution surrounds the target scene (e.g., evenly distributed along the scene boundaries, filling blind spots in the original viewpoint (i.e., the shooting viewpoint of the 3D point cloud to be semantically annotated); b) focusing on covering areas in the original viewpoint that are obstructed by objects (e.g., behind objects, corners, under furniture); c) ultimately achieving the goal of "observing the scene from more dimensions," overcoming the viewpoint limitations of 3D point clouds. For example, the virtual camera's intrinsic parameters can be consistent with those of the real shooting camera; the virtual camera's extrinsic parameters ensure that the virtual camera is distributed around the target scene. Center and radius of the sphere On the surface of a sphere ( (As an empirical coefficient to ensure the virtual camera covers the target scene); azimuth angle in spherical coordinates. Pick (interval pitch angle Pick (interval ), generated in total Individual pose; camera position Convert from spherical coordinates to world coordinates: Rotation matrix Used to establish the camera coordinate system ( Axial forward, pointing Aligned with the world coordinate system, by The direction vector is calculated.

[0050] For step 1013, perform the following operations for each viewpoint: Based on the camera pose of the viewpoint, project the 3D scene model onto the image plane of the virtual camera of the viewpoint to obtain the 2D image corresponding to the viewpoint. Specifically, for each shooting camera pose, use a differentiable rendering process based on 3D Gaussian sputtering technology to project the 3D scene model onto the image plane of the virtual camera of the viewpoint to obtain the 2D image corresponding to the viewpoint. In this way, by generating 2D images of any new viewpoint through 3D Gaussian sputtering technology, the physical occlusion problem is effectively solved, ensuring that all regions in the target scene can be observed and segmented, thereby improving the integrity and accuracy of 3D point cloud semantic annotation; moreover, 3D Gaussian sputtering technology can generate photorealistic images, which have richer texture and lighting details than images generated by traditional mesh-based rendering methods, providing higher quality input for subsequent 2D semantic segmentation models, thus ensuring the accuracy of segmentation from the source.

[0051] Applying the above embodiments, step 1011 constructs an explicit 3D representation of the target scene using 3D GS, achieving accurate scene encoding; step 1012's strategic virtual camera pose covers the occluded areas of the original viewpoint, expanding the scene observation dimensions; step 1013's 3D GS rendering generates a highly realistic 2D image with textures and lighting consistent with the original scene. These three elements work together to overcome the viewpoint limitations of the original data, providing comprehensive and high-quality image input for subsequent multi-view semantic segmentation, ensuring the integrity and accuracy of 3D point cloud semantic annotation.

[0052] In some embodiments, see Figure 5Step 1011, “Constructing a 3D scene model of the target scene”, can be achieved by executing the following steps 201-204: Step 201, acquire 2D scene images of the target scene taken from a first-person perspective, and acquire the camera pose of the camera with the first-person perspective; Step 202, initialize based on the 2D scene images to obtain an initial 3D scene model; Step 203, based on the camera pose, project the initial 3D representation onto the image plane of the camera to obtain a predicted 2D image; Step 204, based on the difference between the predicted 2D image and the 2D scene image, update the initial 3D scene model to obtain a 3D scene model.

[0053] For step 201, obtain a set of two-dimensional images of the target scene taken from multiple different original viewpoints (i.e., first-person viewpoints). The set of camera poses for capturing two-dimensional images of the scene. ,in, It includes camera extrinsic parameters (rotation matrix and translation vector) and intrinsic parameters (focal length and principal point coordinates).

[0054] For step 202, based on 3D Gaussian sputtering (3D GS) technology, an initial 3D scene model of the target scene is initialized, i.e. A set of three-dimensional Gaussian distributions Each Gaussian From parameter set Define, where, The coordinates are in the world coordinate system centered at Gauss. The covariance matrix is ​​calculated from the scaling factor and the rotation quaternion. The opacity is Gaussian. This is Gaussian color information.

[0055] For step 203, based on the camera pose, the initial 3D representation is projected onto the image plane of the camera to obtain the predicted 2D image. That is, for each camera pose... By utilizing the differentiable rendering process of 3D Gaussian sputtering technology, the initial 3D representation is projected onto the image plane of the camera to generate the corresponding predicted 2D image.

[0056] For step 204, a loss function is predefined; then, the difference between the predicted 2D image and the scene 2D image is calculated, and the value of the loss function is determined based on this difference. A gradient descent optimization algorithm (such as the Adam algorithm) is then used to iteratively update the parameters (i.e., the parameter set) of the initial 3D scene model, with the objective of minimizing the loss function value (i.e., minimizing the difference). The iteration terminates when the iteration termination condition is met (such as the value of the loss function being less than a threshold), thus obtaining an optimized 3D scene model.

[0057] Applying the above embodiments, step 201 provides a ground truth benchmark (i.e., a 2D scene image) for modeling the 3D scene using real-world captured data; step 202 initializes the basic structure of the 3D scene (i.e., the initial 3D scene model), providing a starting point for subsequent optimization; step 203 generates a predicted 2D image through projection; and step 204 significantly improves the accuracy of the 3D scene model by optimizing the difference between the scene's 2D image and the predicted 2D image (the Gaussian parameters are consistent with the geometry and appearance of the real scene). Ultimately, the optimized 3D scene model can accurately encode the 3D structure and texture of the target scene, providing a high-fidelity underlying foundation for subsequent steps such as "multi-view virtual image generation" and "semantic segmentation."

[0058] Step 102: For each two-dimensional image, determine the first semantic label of each pixel in the two-dimensional image.

[0059] Step 102 involves converting the visual information of the two-dimensional images corresponding to each viewpoint into semantic information (i.e., the first semantic label L(u,v) of each pixel point (u,v)), which provides a pixel-level semantic foundation for the subsequent mapping of semantic labels to 3D point clouds and is a key bridge connecting two-dimensional images and three-dimensional point clouds.

[0060] In some embodiments, the step 102 of "determining the first semantic label of each pixel in the two-dimensional image" can be achieved by performing the following steps: obtaining the descriptive text of the two-dimensional image, the descriptive text being used to describe the target object in the two-dimensional image; encoding the two-dimensional image to obtain image features, and encoding the descriptive text to obtain text features; fusing the image features and text features to obtain fused features; and predicting the first semantic label of each pixel using a machine learning model based on the fused features.

[0061] Here, the descriptive text used to describe the target objects in the 2D image is first obtained. The descriptive text is a natural language phrase or sentence, and each phrase corresponds to a target semantic category (such as "red fabric sofa", "off-white latex paint wall", "light brown wooden floor", "pedestrian", "wall", "ground", etc.). The descriptive text is input by the user according to the semantic annotation requirements of the target scene, or selected from a preset semantic category library (the category library contains standardized descriptions of common scene objects).

[0062] Then, the two-dimensional image is encoded to obtain image features, and the descriptive text is encoded to obtain text features. Specifically, the encoding process for the two-dimensional image includes: encoding the two-dimensional image using a pre-trained visual Transformer model (such as VisionTransformer, ViT); assuming the two-dimensional image is... ,Will Divided into Image patches ( Image height, Image width, For block size, such as Each image patch (pixels) is flattened into a vector and input into the ViT model. The ViT model processes the image patches through a multi-layer self-attention mechanism and a feedforward neural network, outputting image features. The encoding process for the descriptive text includes: encoding the descriptive text using a pre-trained text Transformer model (such as the CLIP text encoder); let the descriptive text be... ( (Number of semantic categories) per text phrase The input text encoder, after word embedding, position encoding and multi-layer self-attention processing, outputs text features.

[0063] The image features and text features are then fused to obtain the fused features. Specifically, a cross-modal attention mechanism can be used to fuse image features and text features to enhance the semantic relevance of image features. For example, through the attention mechanism, for each image patch feature in the image features, the semantic relevance between that image patch feature and the text feature (i.e., the attention weight of that image patch feature and the text feature) is calculated. Then, based on the semantic relevance of each image patch feature, the image patch features are weighted to obtain weighted image patch features. The weighted image patch features are then added to the original image patch features to obtain the target image patch features. Finally, all target image patch features are integrated to obtain the fused features.

[0064] Finally, based on the fused features, a machine learning model predicts the first semantic label for each pixel. In practical applications, this machine learning model is a text-image segmentation model. It predicts the probability of each pixel belonging to each semantic label based on the fused features, and then uses the semantic label with the highest probability as the first semantic label. Thus, by employing a text-segmentation model based on text prompts, arbitrary semantic categories can be flexibly defined according to requirements without retraining the text-image segmentation model, greatly improving the versatility and automation level of this application.

[0065] By applying the above embodiments, semantic categories are precisely defined by describing text, avoiding the dependence of traditional semantic segmentation on pre-trained category libraries. Cross-modal attention is used to fuse image visual features and text semantic features, enhancing the semantic directionality of image features and effectively solving the confusion problem of "visually similar but semantically different" (such as "red sofa" and "red cushion"). The final predicted first semantic label combines the appearance information of the image with the semantic constraints of the text, significantly improving the accuracy of pixel-level semantic segmentation and providing a highly reliable two-dimensional semantic foundation for subsequent 3D point cloud semantic annotation.

[0066] Step 103: For each data point in the 3D point cloud of the target scene, detect the visibility of the data points from multiple perspectives and obtain the detection results.

[0067] For step 103, the 3D point cloud of the target scene is the point cloud for which semantic determination of the point cloud is to be performed. The 3D point cloud includes multiple data points. First, the following processing is performed on each data point: the visibility of the data point is detected from multiple viewpoints, and the detection results are obtained. Specifically, the visibility of the data point is detected from each viewpoint, and the detection results for each viewpoint are obtained. Visibility detection refers to: detecting whether the line connecting the data point and the camera center of the virtual camera is occluded by other geometric objects in the target scene under the viewpoint of the virtual camera, and whether the depth value of the data point (i.e., the distance from the data point to the camera center) satisfies the set conditions with the depth value of the corresponding pixel in the depth map of that viewpoint (i.e., detecting whether the data point is within the camera's view frustum). This will be explained in detail below.

[0068] In some embodiments, see Figure 6 In step 103, "detecting the visibility of data points from multiple viewpoints and obtaining detection results" can be achieved by performing the following steps 1031-1034 for each viewpoint: Step 1031, determine the center point of the virtual camera with the viewpoint and determine the distance between the data point and the camera center point; Step 1032, obtain the depth value of the pixel corresponding to the data point in the two-dimensional image of the viewpoint and determine the absolute value of the difference between the distance and the depth value; Step 1033, if the absolute value is less than a preset threshold, determine that a first detection result is obtained, which indicates that the data point is visible in the viewpoint; Step 1034, if the absolute value is greater than or equal to the preset threshold, determine that a second detection result is obtained, which indicates that the data point is not visible in the viewpoint.

[0069] Here, the test results include the first test result and the second test result.

[0070] For step 1031, the camera center point is the geometric center of the camera in the world coordinate system, corresponding to the translation vector (camera position) in the extrinsic parameters. It is the "perspective center" of the 3D points projected onto the image plane (all projection lines pass through this point), and is the core reference point for calculating the projection positions (projection points) of data points in the 3D point cloud. After determining the camera center point, based on the coordinates of the camera center point (… ) and coordinates of data points ( ), calculate the distance (such as Euclidean distance) between the data point and the camera center point, which is the depth value of the data point in the view of the virtual camera.

[0071] For step 1032, the depth value of the pixel corresponding to the data point in the two-dimensional image of the viewpoint is obtained. In practical applications, while rendering the two-dimensional image of each viewpoint based on the 3D scene model using 3D Gaussian sputtering technology, a depth map corresponding to each two-dimensional image is also rendered. This depth map includes the depth of each pixel in the two-dimensional image. The data point can be projected onto the image plane of the virtual camera to obtain the coordinates of the projection point; the pixel in the two-dimensional image with the same coordinates as the projection point is taken as the pixel corresponding to the data point. At this time, the depth value of the pixel is extracted from the depth map based on the coordinates of the projection point (since the two are the same, they are also the pixel coordinates). Then, the absolute value of the difference between the distance and the depth value is calculated.

[0072] For step 1033, if the absolute value is less than the preset threshold, the first detection result is determined, and the first detection result indicates that the data point is visible from the viewpoint.

[0073] For step 1034, if the absolute value is greater than or equal to the preset threshold, the second detection result is determined. The second detection result represents the data point that is not visible from the viewpoint.

[0074] By applying the above embodiments, the visibility of data points is accurately determined by calculating the true distance between the data points and the camera center point and comparing it with the depth value of the virtual rendering. This is a key filtering condition for multi-view semantic fusion. By retaining only the first semantic label of the visible target viewpoint, it avoids erroneous interference from invisible viewpoints, significantly improving the accuracy and robustness of 3D point cloud semantic annotation.

[0075] Step 104: For each data point, if the detection result indicates that the target viewpoint of the data point is visible in multiple viewpoints, determine the target two-dimensional image corresponding to the target viewpoint from multiple two-dimensional images, determine the first pixel point corresponding to the projection point of the data point in the target viewpoint from each pixel point of the target two-dimensional image, and use the first semantic label of the first pixel point as the target first semantic label of the data point.

[0076] For step 104, the following processing is performed for each data point: If the detection result indicates that the data point is visible in the target view (at least one target view) across multiple viewpoints, then firstly, the target 2D image corresponding to the target viewpoint is determined from the 2D images corresponding to the multiple viewpoints. Then, from each pixel of the target 2D image, the first pixel corresponding to the projection point of the data point in the target viewpoint is determined. The first semantic label of the first pixel is then used as the target first semantic label of the data point in the target viewpoint. Thus, the semantic label of the data point in the visible target viewpoint is obtained.

[0077] In some embodiments, see Figure 7Before executing step 104, "determine the target first semantic label of the data point in the target view from each first semantic label of the two-dimensional image corresponding to the target view", the following steps 301-302 can also be executed: Step 301, determine the target camera pose of the target virtual camera with the target view; Step 302, based on the target camera pose, project the data point onto the image plane of the target virtual camera to obtain the projection point of the data point in the target view.

[0078] For step 301, the target camera pose of the target virtual camera with the target viewpoint is obtained. The target camera pose includes extrinsic parameters (rotation matrix and translation vector) and intrinsic parameters (focal length and principal point coordinates). Among them, the rotation matrix is ​​used to transform the world coordinate system into the camera coordinate system of the target virtual camera; the translation vector is the coordinate of the camera center point of the target virtual camera in the world coordinate system.

[0079] For step 302, based on the target camera pose, the data points are projected onto the image plane of the target virtual camera to obtain the projected points (specifically, the coordinates of the projected points). Specifically, based on the target camera pose, the data points are first converted from the world coordinate system to the camera coordinate system of the target virtual camera, and then the data points in the camera coordinate system are projected onto the image plane of the target virtual camera to obtain the coordinates of the projected points.

[0080] Based on steps 301-302, determining the first pixel corresponding to the projection point includes: mapping the projection point coordinates to the pixel coordinate system of the two-dimensional image corresponding to the target viewpoint, obtaining the target projection point coordinates in the pixel coordinate system; rounding the target projection point coordinates to obtain the rounded coordinates; and using the pixel indicated by the rounded coordinates in the two-dimensional image as the first pixel. From the first semantic labels of each pixel in the two-dimensional image, extracting the first semantic label corresponding to the first pixel, and using the first semantic label corresponding to the first pixel as the target first semantic label.

[0081] Applying the above embodiments, the spatial association between 3D data points and 2D semantic labels is achieved through the process of "pose determination - projection mapping - label extraction": the target camera pose in step 301 ensures the viewpoint coverage of the virtual camera; the projection calculation in step 302 accurately establishes the correspondence between 3D data points and 2D image pixels; based on this, "using the first semantic label of the first pixel as the target first semantic label of the data point" directly maps the 2D semantic segmentation result to the 3D data point. Its core value lies in providing accurate semantic information from a single viewpoint for subsequent "multi-view semantic fusion," avoiding the spatial misalignment problem between semantic labels and 3D point clouds, ensuring that the semantic label of each 3D data point is consistent with its visual features under a specific viewpoint, and laying the foundation for the accuracy of the final 3D point cloud semantic annotation.

[0082] Step 105: For each data point, determine the second semantic label of the data point based on the target first semantic label of the data point.

[0083] For step 105, for each data point, the first semantic label of the data point is collected from all target viewpoints (i.e., the first semantic label corresponding to the projection of the data point onto the 2D image from a single viewpoint). These first semantic labels are then fused across all target viewpoints to obtain the second semantic label of the data point. For example, a preset fusion strategy (such as majority voting or confidence weighting) can be used to calculate the score (e.g., number of votes or weighting value) of each target's first semantic label; the target's first semantic label with the highest score is then used as the second semantic label for that data point. The fusion strategy aims to integrate information from multiple target viewpoints, suppress occlusion, projection errors, or semantic segmentation errors from a single viewpoint, and ensure that the second semantic label is closer to the true semantics of the data point, thereby improving the accuracy, robustness, and completeness of semantic annotation.

[0084] In some embodiments, step 105, "determining the second semantic label of a data point based on the first semantic label of the target pixel corresponding to the data point," can be achieved by performing the following steps: if the number of target views is one, the first semantic label of the target is determined as the second semantic label; if the number of target views is multiple and the first semantic label of the multiple target views is unique, the first semantic label of the target is determined as the second semantic label; if the number of target views is multiple and the first semantic labels of the multiple target views are not unique, for each target first semantic label, the number of target views corresponding to the first semantic label of the target is obtained, and the first semantic label of the target with the largest number is determined as the second semantic label.

[0085] (1) If there is only one target viewpoint, then the first semantic label of the target is determined as the second semantic label.

[0086] (2) If there are multiple target viewpoints and the target first semantic label is unique across multiple target viewpoints, the target first semantic label is determined as the second semantic label. Here, the target first semantic label determined by the data point in multiple target viewpoints is the same. Therefore, the same unique target first semantic label is determined as the second semantic label.

[0087] (3) If there are multiple target viewpoints and the target first semantic labels of the multiple target viewpoints are not unique, that is, the target first semantic labels determined by the data points under multiple target viewpoints are not completely the same (i.e., there are at least two different target first semantic labels), then for each different target first semantic label, the number of target viewpoints corresponding to the target first semantic label is counted, and the target first semantic label corresponding to the largest number is determined as the second semantic label. In practical applications, initial voting scores (e.g., 0) can be set for each of the multiple semantic labels (including the first semantic label) in advance; when performing visibility detection, whenever a data point is detected to be visible in a target viewpoint, the voting score corresponding to the target first semantic label of the data point in that target viewpoint is increased by 1 (or increased by 1 * the confidence of the target first semantic label); in this way, the voting scores corresponding to each different target first semantic label are obtained, and the target first semantic label with the highest voting score is selected as the second semantic label.

[0088] Applying the above embodiments: 1) For scenarios where the semantic labels of a single target view or multiple target views are consistent, the first semantic label of the target is directly reused as the second semantic label, avoiding redundant calculations and improving processing efficiency. 2) For scenarios where the semantic labels of multiple target views are inconsistent, the number of target views corresponding to the semantic labels is counted and the one with the most is selected. The information from multiple target views is used to suppress occlusion, projection errors, or semantic segmentation errors of a single view, ensuring that the second semantic label is closer to the true semantics of the data points. 3) The rule logic is simple and highly adaptable, covering scenarios with different numbers of views, effectively improving the accuracy and robustness of 3D point cloud semantic annotation.

[0089] By applying the embodiments described above, multi-view 2D images of the target scene are generated to cover occluded areas that cannot be observed from a single viewpoint. First semantic labels are determined for pixels in the 2D images from each viewpoint, providing a foundation for multi-view point cloud semantic determination. The visibility of data points in the 3D point cloud across multiple views is detected, and visible target views are selected to avoid interference from incorrect semantic labels from invisible views. Then, second semantic labels are determined based on the first semantic labels of data points in visible target views, achieving multi-view point cloud semantic determination. This effectively solves the problem of semantic determination errors caused by single-view occlusion and improves the accuracy of 3D point cloud semantic segmentation.

[0090] The following description continues to illustrate the exemplary structure of the point cloud data processing apparatus 555 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2As shown, the software modules in the point cloud data processing device 555 stored in the memory 550 may include: a generation module 5551, used to generate two-dimensional images of the target scene corresponding to multiple viewpoints respectively; a first determination module 5552, used to determine a first semantic label for each pixel in the two-dimensional image for each of the two-dimensional images; a detection module 5553, used to detect the visibility of each data point in the three-dimensional point cloud of the target scene in the multiple viewpoints, and obtain a detection result; a second determination module 5554, used to determine a target two-dimensional image corresponding to the target viewpoint from the multiple two-dimensional images if the detection result indicates that the data point is visible in the target viewpoint in the multiple viewpoints, and determine a first pixel corresponding to the projection point of the data point in the target viewpoint from each pixel in the target two-dimensional image, and use the first semantic label of the first pixel as the target first semantic label of the data point; and a third determination module 5555, used to determine a second semantic label of the data point based on the target first semantic label of the data point for each of the data points.

[0091] In some embodiments, the generation module 5551 is further configured to construct a three-dimensional scene model of the target scene; determine the camera pose of the virtual camera with the viewpoint for each viewpoint; and project the three-dimensional scene model onto the image plane of the virtual camera of the viewpoint based on the camera pose of the viewpoint for each viewpoint to obtain the two-dimensional image corresponding to the viewpoint.

[0092] In some embodiments, the generation module 5551 is further configured to acquire two-dimensional images of the target scene taken from a first perspective, and acquire the camera pose of the camera with the first perspective; initialize based on the two-dimensional images of the scene to obtain an initial three-dimensional scene model; project the initial three-dimensional representation onto the image plane of the camera based on the camera pose to obtain a predicted two-dimensional image; and update the initial three-dimensional scene model based on the difference between the predicted two-dimensional image and the two-dimensional images of the scene to obtain the three-dimensional scene model.

[0093] In some embodiments, the first determining module 5552 is further configured to obtain descriptive text of the two-dimensional image, the descriptive text being used to describe a target object in the two-dimensional image; encode the two-dimensional image to obtain image features, and encode the descriptive text to obtain text features; fuse the image features and the text features to obtain fused features; and based on the fused features, predict the first semantic label of each pixel using a machine learning model.

[0094] In some embodiments, the detection module 5553 is further configured to perform the following processing for each of the viewpoints: determine the camera center point of the virtual camera having the viewpoint, and determine the distance between the data point and the camera center point; obtain the depth value of the pixel corresponding to the data point in the two-dimensional image of the viewpoint, and determine the absolute value of the difference between the distance and the depth value; if the absolute value is less than a preset threshold, determine a first detection result, the first detection result indicating that the data point is visible in the viewpoint; if the absolute value is greater than or equal to the preset threshold, determine a second detection result, the second detection result indicating that the data point is not visible in the viewpoint, the detection result including the first detection result and the second detection result.

[0095] In some embodiments, the second determining module 5554 is further configured to determine the target camera pose of the target virtual camera having the target viewpoint before determining the first pixel point corresponding to the projection point of the data point in the target viewpoint from each pixel point of the target two-dimensional image; and based on the target camera pose, project the data point onto the image plane of the target virtual camera to obtain the projection point of the data point in the target viewpoint.

[0096] In some embodiments, the third determining module 5555 is further configured to: if the number of target viewpoints is one, determine the target first semantic tag as the second semantic tag; if the number of target viewpoints is multiple and the target first semantic tags of the multiple target viewpoints are unique, determine the target first semantic tag as the second semantic tag; if the number of target viewpoints is multiple and the target first semantic tags of the multiple target viewpoints are not unique, for each target first semantic tag, obtain the number of target viewpoints corresponding to the target first semantic tag, and determine the target first semantic tag corresponding to the largest number as the second semantic tag.

[0097] It should be noted that the description of the device embodiments in this application is similar to the description of the method embodiments described above, and has similar beneficial effects as the method embodiments, so it will not be repeated here. Any technical details not covered in the point cloud data processing device provided in the embodiments of this application can be understood based on the description of the technical details in the above method embodiments.

[0098] This application also provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the point cloud data processing method provided in this application.

[0099] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the point cloud data processing method provided in this application.

[0100] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0101] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0102] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0103] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0104] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for processing point cloud data, characterized in that, The method includes: Generate two-dimensional images of the target scene from multiple perspectives; For each of the two-dimensional images, determine the first semantic label for each pixel in the two-dimensional image; For each data point in the 3D point cloud of the target scene, the visibility of the data points in multiple viewpoints is detected to obtain the detection results; For each of the data points, if the detection result indicates that the data point is visible in the target view of the multiple viewpoints, the target two-dimensional image corresponding to the target viewpoint is determined from the multiple two-dimensional images, and the first pixel point corresponding to the projection point of the data point in the target viewpoint is determined from each pixel point of the target two-dimensional image, and the first semantic label of the first pixel point is used as the target first semantic label of the data point. For each data point, a second semantic label is determined based on the target first semantic label of the data point.

2. The method as described in claim 1, characterized in that, The generated target scene comprises two-dimensional images corresponding to multiple viewpoints, including: Construct a three-dimensional scene model of the target scene; For each of the aforementioned viewpoints, determine the camera pose of the virtual camera that has the aforementioned viewpoint; For each of the aforementioned viewpoints, based on the camera pose of the viewpoint, the 3D scene model is projected onto the image plane of the virtual camera of the viewpoint to obtain the 2D image corresponding to the viewpoint.

3. The method as described in claim 2, characterized in that, The construction of the 3D scene model of the target scene includes: Acquire two-dimensional images of the target scene taken from a first perspective, and acquire the camera pose of the camera with the first perspective. An initial three-dimensional scene model is obtained by initializing based on the two-dimensional image of the scene. Based on the camera pose, the initial 3D representation is projected onto the image plane of the camera to obtain a predicted 2D image; Based on the difference between the predicted 2D image and the scene 2D image, the initial 3D scene model is updated to obtain the 3D scene model.

4. The method as described in claim 1, characterized in that, Determining the first semantic label for each pixel in the two-dimensional image includes: Obtain the descriptive text of the two-dimensional image, wherein the descriptive text is used to describe the target object in the two-dimensional image; The two-dimensional image is encoded to obtain image features, and the descriptive text is encoded to obtain text features; The image features and the text features are fused to obtain fused features; Based on the fusion features, the first semantic label of each pixel is predicted by a machine learning model.

5. The method as described in claim 1, characterized in that, The detection of the visibility of the data points from multiple viewpoints to obtain detection results includes: Perform the following processing for each of the aforementioned perspectives: Determine the center point of the virtual camera with the stated viewpoint, and determine the distance between the data point and the center point of the camera; Obtain the depth value of the pixel corresponding to the data point in the two-dimensional image of the viewpoint, and determine the absolute value of the difference between the distance and the depth value; If the absolute value is less than a preset threshold, a first detection result is obtained, and the first detection result indicates that the data point is visible in the viewpoint. If the absolute value is greater than or equal to the preset threshold, a second detection result is obtained. The second detection result indicates that the data point is not visible in the viewpoint. The detection result includes the first detection result and the second detection result.

6. The method as described in claim 1, characterized in that, The method of determining the data point before the first pixel corresponding to the projection point of the target viewpoint from each pixel of the target two-dimensional image further includes: Determine the target camera pose of the target virtual camera with the target viewpoint; Based on the target camera pose, the data points are projected onto the image plane of the target virtual camera to obtain the projection point of the data points in the target viewpoint.

7. The method as described in claim 1, characterized in that, Determining the second semantic label of the data point based on the target first semantic label of the data point includes: If the number of target viewpoints is one, the first semantic label of the target is determined as the second semantic label; If there are multiple target viewpoints and the target first semantic label of the multiple target viewpoints is unique, then the target first semantic label is determined as the second semantic label; If there are multiple target viewpoints and the target first semantic labels of the multiple target viewpoints are not unique, for each target first semantic label, obtain the number of target viewpoints corresponding to the target first semantic label, and determine the target first semantic label corresponding to the largest number as the second semantic label.

8. A point cloud data processing device, characterized in that, The device includes: The generation module is used to generate two-dimensional images of the target scene from multiple viewpoints. The first determining module is used to determine the first semantic label of each pixel in the two-dimensional image for each of the two-dimensional images; The detection module is used to detect the visibility of each data point in the 3D point cloud of the target scene from multiple viewpoints and obtain the detection results. The second determining module is used for each of the data points. If the detection result indicates that the data point is visible in the target view of the multiple viewpoints, it determines the target two-dimensional image corresponding to the target viewpoint from the multiple two-dimensional images, determines the first pixel point corresponding to the projection point of the data point in the target viewpoint from each pixel point of the target two-dimensional image, and uses the first semantic label of the first pixel point as the target first semantic label of the data point. The third determining module is used to determine the second semantic label of each data point based on the target first semantic label of the data point.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the point cloud data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the point cloud data processing method according to any one of claims 1 to 7 is implemented.

Images