Point cloud attribute assignment method for depth camera, system and electronic device thereof

Abstract

A method for assigning point cloud attributes to a depth camera, a system for assigning point cloud attributes to a depth camera, and an electronic device thereof. The method comprises the following steps: performing target detection processing on two-dimensional image data acquired by the depth camera to obtain a target detection result, wherein the target detection result includes a target frame, a target category, and corresponding category attributes of one or more detected targets; performing point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain a target point cloud area; and assigning the category attribute of the detected target to the corresponding target point cloud area.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional vision technology, and in particular to a point cloud attribute assignment method for a depth camera, a system thereof, and an electronic device. Background Art

[0002] In recent years, three-dimensional vision technology has developed rapidly, and point cloud processing technology is the foundation of the development of three-dimensional vision technology. A point cloud is a massive collection of points that expresses the spatial distribution and surface characteristics of a target in the same spatial reference system. Combining the principles of laser measurement and photogrammetry, point cloud attribute information includes three-dimensional coordinates (XYZ), laser reflection intensity (Intensity), and color information (RGB). However, with the development of path planning, intelligent obstacle avoidance, target following and other technologies in robot vision technology, traditional three-dimensional coordinates, laser reflection intensity and color information can no longer meet the needs. Point clouds need to be assigned more attributes, such as object category attributes, to adapt to different scene requirements.

[0003] Currently, existing 3D point cloud object recognition methods mostly extract the geometric attributes, shape attributes, structural attributes, or a combination of multiple attributes of the object's feature points, compare and learn the features, and thus complete the object recognition and classification. However, with the rapid development of computer vision and cognitive technologies, the recognition of objects in a scene has gradually transitioned from the recognition of specific objects (such as vehicles, trees, or buildings) to the recognition of multiple types of objects. This relies on complex pre-processing of large amounts of 3D point cloud data in advance. Because many point cloud data are not open source, the required cost is relatively high, making the existing 3D point cloud object recognition methods difficult to promote and popularize. Summary of the Invention

[0004] One advantage of the present invention is that it provides a point cloud attribute assignment method for a depth camera, a system thereof, and an electronic device. Unlike existing three-dimensional recognition methods, it does not require complex target object preprocessing of a large amount of point cloud data in advance. Instead, it uses a large amount of existing open source two-dimensional image information as a data source to complete the assignment of point cloud attributes, which helps to significantly reduce the required costs and facilitates promotion and popularization.

[0005] Another advantage of the present invention is that it provides a point cloud attribute assignment method for a depth camera, a system thereof, and an electronic device. In one embodiment of the present invention, the point cloud attribute assignment method for a depth camera can freely assign attributes to the point cloud, so that the point cloud adds more intelligent attributes on the basis of existing basic attributes to adapt to more complex scenes.

[0006] Another advantage of the present invention is that it provides a point cloud attribute assignment method for a depth camera, a system thereof, and an electronic device. In one embodiment of the present invention, the point cloud attribute assignment method for a depth camera can utilize the texture information of the RGB image for target detection, and utilize the spatial information of the point cloud and the RGB recognition results for point cloud segmentation and attribute assignment, thereby solving the current problem of insufficient and difficult three-dimensional recognition annotation data.

[0007] Another advantage of the present invention is that it provides a point cloud attribute assignment method for a depth camera, a system therefor, and an electronic device. In one embodiment of the present invention, the point cloud attribute assignment method for a depth camera can utilize a deep learning target detection method, so that the assignment of point cloud attributes is more accurate and faster, and the category definition is richer and more flexible, which helps to meet the needs of different scenarios.

[0008] Another advantage of the present invention is that it provides a point cloud attribute assignment method for a depth camera, a system thereof, and an electronic device. In one embodiment of the present invention, the point cloud attribute assignment method for a depth camera can quickly find the corresponding point cloud area and the RGB information corresponding to the point cloud through the RGBD mapping method, so as to make full use of color information to assist spatial information, making point cloud segmentation more accurate.

[0009] Another advantage of the present invention is that it provides a point cloud attribute assignment method for a depth camera, a system thereof, and an electronic device. In one embodiment of the present invention, the point cloud attribute assignment method for a depth camera can use methods such as connected domain filtering to remove part of the point cloud noise, and use methods such as ground detection and dynamic histogram to segment the ground, foreground and background, thereby obtaining a point cloud block that accurately corresponds to the target detection result.

[0010] Another advantage of the present invention is providing a method for assigning point cloud attributes to a depth camera, a system therefor, and an electronic device therefor. To achieve these advantages, the present invention does not require complex structures or extensive computational effort, and has low hardware and software requirements. Therefore, the present invention successfully and effectively provides a solution that not only provides a method for assigning point cloud attributes to a depth camera, a system therefor, and an electronic device therefor, but also increases the practicality and reliability of the method, system therefor, and electronic device therefor.

[0011] In order to achieve at least one of the above advantages or other advantages and purposes, the present invention provides a point cloud attribute assignment method for a depth camera, comprising the steps of:

[0012] Performing target detection processing on the two-dimensional image data acquired by the depth camera to obtain a target detection result, wherein the target detection result includes a target frame, a target category, and corresponding category attributes of one or more detected targets;

[0013] performing point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain a target point cloud area; and

[0014] Assign the category attribute of the detected target to the corresponding target point cloud area.

[0015] According to an embodiment of the present invention, the depth camera is used to collect RGBD data, wherein the two-dimensional image data is RGB data.

[0016] According to one embodiment of the present invention, the step of performing target detection processing on the two-dimensional image data acquired by the depth camera to obtain a target detection result, wherein the target detection result includes a target frame, a target category, and a corresponding category attribute of one or more detected targets, includes the steps of:

[0017] Detecting the RGB data using a deep learning-based target detection model to obtain the target frame and the target category of the detected target; and

[0018] By freely defining the category of the detection target based on the open source target box annotation model, the category attributes of the detection target are obtained.

[0019] According to one embodiment of the present invention, the step of performing point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain the target point cloud area includes the steps of:

[0020] The point cloud data is preliminarily segmented through the RGBD mapping model to obtain the initial point cloud area corresponding to the target box of the detection target;

[0021] Perform ground detection on the initial point cloud area to obtain a ground-removed point cloud area;

[0022] Perform dynamic histogram statistics on the ground-removed point cloud area to obtain the number of point clouds in each bin interval; and

[0023] Determine whether the maximum bin interval and the second largest bin interval are adjacent. If so, select two continuous intervals corresponding to the maximum bin interval and the second largest bin interval respectively as the target point cloud area; if not, select a continuous area corresponding to the maximum bin interval as the target point cloud area.

[0024] According to one embodiment of the present invention, the step of performing point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain the target point cloud area, before the step of performing ground detection on the initial point cloud area to obtain the ground-removed point cloud area, further includes the steps of:

[0025] The initial point cloud region is subjected to point cloud preprocessing by connected domain filtering to obtain a denoised point cloud region, so that ground detection is subsequently performed on the denoised point cloud region to obtain the ground-deleted point cloud region.

[0026] According to an embodiment of the present invention, the depth camera is a TOF camera.

[0027] According to another aspect of the present invention, the present invention further provides a point cloud attribute assignment system for a depth camera, comprising:

[0028] a detection module, configured to perform target detection processing on the two-dimensional image data acquired by the depth camera to obtain a target detection result, wherein the target detection result includes a target frame, target category, and corresponding category attributes of one or more detected targets;

[0029] a segmentation module for performing point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain a target point cloud area; and

[0030] An assignment module is used to assign the category attribute of the detected target to the corresponding target point cloud area.

[0031] According to an embodiment of the present invention, the depth camera is used to collect RGBD data, wherein the two-dimensional image data is RGB data.

[0032] According to one embodiment of the present invention, the detection module includes: a target detection module, which is communicatively connected to each other, and is used to detect the RGB data through a deep learning-based target detection model to obtain the target box and the target category of the detected target; and a target labeling module, which is used to freely define the category of the detected target through an open source target box labeling model to obtain the category attribute of the detected target.

[0033] According to one embodiment of the present invention, the segmentation module includes: a preliminary segmentation module, which is communicatively connected to each other, and is used to perform preliminary segmentation on the point cloud data through an RGBD mapping model to obtain an initial point cloud area corresponding to the target box of the detection target; a ground detection module, which is used to perform ground detection on the initial point cloud area to obtain a point cloud area removed from the ground; a histogram statistics module, which is used to perform dynamic histogram statistics on the point cloud area removed from the ground to obtain the number of point clouds in each bin interval; and a judgment and selection module, which is used to judge whether the maximum bin interval and the second largest bin interval are adjacent. If so, two continuous intervals corresponding to the maximum bin interval and the second largest bin interval are selected as the target point cloud area; if not, a continuous area corresponding to the maximum bin interval is selected as the target point cloud area.

[0034] According to one embodiment of the present invention, the segmentation module further includes a preprocessing module, wherein the preprocessing module is used to perform point cloud preprocessing on the initial point cloud area through connected domain filtering to obtain a denoised point cloud area, so that the denoised point cloud area is subsequently subjected to ground detection to obtain the denoised point cloud area.

[0035] According to another aspect of the present invention, the present invention further provides an electronic device, comprising:

[0036] at least one processor for executing instructions; and

[0037] A memory communicatively connected to the at least one processor, wherein the memory has at least one instruction, wherein the instruction is executed by the at least one processor to cause the at least one processor to perform some or all steps of a point cloud attribute assignment method for a depth camera, wherein the point cloud attribute assignment method for a depth camera comprises the steps of:

[0038] Performing target detection processing on the two-dimensional image data acquired by the depth camera to obtain a target detection result, wherein the target detection result includes a target frame, a target category, and corresponding category attributes of one or more detected targets;

[0039] performing point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain a target point cloud area; and

[0040] Assign the category attribute of the detected target to the corresponding target point cloud area.

[0041] Further objects and advantages of the present invention will be fully apparent from an understanding of the following description and accompanying drawings.

[0042] These and other objects, features and advantages of the present invention will be more fully understood from the following detailed description, accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 4 is a flow chart of a method for assigning point cloud attributes to a depth camera according to an embodiment of the present invention.

[0044] Figure 2 A flow chart illustrating one of the steps of the point cloud attribute assignment method for a depth camera according to the above embodiment of the present invention is shown.

[0045] Figure 3 A flow chart of step 2 of the point cloud attribute assignment method for a depth camera according to the above embodiment of the present invention is shown.

[0046] Figure 4 A schematic diagram is shown of RGB information acquired by a depth camera and corresponding target detection results according to the above embodiment of the present invention.

[0047] Figure 5 A schematic diagram of point cloud data acquired by a depth camera according to the above embodiment of the present invention is shown.

[0048] Figure 6 A schematic diagram of a point cloud after attribute assignment of the point cloud attribute assignment method for a depth camera according to the above embodiment of the present invention is shown.

[0049] Figure 7 1 is a block diagram of a point cloud attribute assignment system for a depth camera according to an embodiment of the present invention.

[0050] Figure 8 A block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0051] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0052] In the present invention, the term "a" or "an" in the claims and the specification should be understood as "one or more." That is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple. Unless the disclosure of the present invention clearly indicates that the number of the element is only one, the term "a" or "an" should not be understood as a unique or singular element, and the term "a" or "an" should not be understood as a limitation on the quantity.

[0053] In the description of the present invention, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through a medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0054] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0055] Currently, existing 3D point cloud object recognition methods mostly extract features such as geometric properties, shape properties, structural properties, or a combination of multiple properties from the object's feature points, and then compare and learn them to complete object recognition and classification. However, with the rapid development of computer vision and cognitive technologies, the recognition of objects in a scene has gradually transitioned from identifying specific objects (such as vehicles, trees, buildings, etc.) to identifying multiple types of objects. This relies on complex preprocessing of large amounts of 3D point cloud data in advance, which is relatively costly. As a result, much point cloud data is not open source.

[0056] Therefore, in order to solve the above problems, this application provides a point cloud attribute assignment method for a depth camera, which can be combined with the current mainstream deep learning-based target detection algorithm, does not require complex target object preprocessing of a large amount of point cloud data in advance, and uses a large amount of existing open source two-dimensional image information as a data source to obtain a variety of target categories and then obtain the corresponding point cloud target category attributes. It is understandable that the depth camera of this application can be implemented as, but is not limited to, a camera that can obtain two-dimensional image data and point cloud data, such as a TOF camera or a structured light camera.

[0057] Schematic method

[0058] Reference to the accompanying drawings Figures 1 to 3 As shown in FIG, a method for assigning point cloud attributes to a depth camera according to an embodiment of the present invention is described. Specifically, Figure 1 As shown, the point cloud attribute assignment method for a depth camera may include the steps of:

[0059] S100: performing target detection processing on the two-dimensional image data acquired by the depth camera to obtain a target detection result, wherein the target detection result includes a target frame, a target category, and corresponding category attributes of one or more detected targets;

[0060] S200: performing point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain a target point cloud area; and

[0061] S300: Assigning the category attribute of the detected target to the corresponding target point cloud area.

[0062] It is worth noting that the point cloud attribute assignment method for depth cameras in this application does not require complex pre-processing of a large amount of three-dimensional power data in advance as in existing three-dimensional point cloud object recognition methods. Instead, it first uses a large amount of existing open-source two-dimensional image information as a data source to perform target detection processing to obtain target frames, target categories, and corresponding category attributes for a variety of detection targets; then, the two-dimensional to three-dimensional mapping relationship of the depth camera is used to easily obtain the target point cloud area, and then the corresponding analog attributes are directly assigned to the target point cloud area. Therefore, the point cloud attribute assignment method for depth cameras in this application can adapt to more application scenarios and significantly reduce the required cost, making it easier to promote and popularize.

[0063] In addition, this application proposes the concept of point cloud attribute assignment for the first time. This allows for the free assignment of attributes to point clouds, such as object category attributes. This allows point clouds to add more intelligent attributes on top of existing basic attributes, allowing for more complex scenarios. For example, in the process of AI intelligent obstacle avoidance, flexible obstacle avoidance methods can be set based on different object attributes. For example, obstacle avoidance at different distances can be achieved based on whether a dynamic or static object is detected.

[0064] Preferably, the two-dimensional image data obtained by the depth camera can be implemented as RGB data, so as to make full use of a large amount of existing open source RGB image information as a data source. In other words, the depth camera of the present application can be implemented as, but not limited to, an RGBD camera for collecting RGBD data, wherein the RGBD data includes RGB data (such as Figure 4 as shown) and point cloud data (as Figure 5 As shown). In this way, the point cloud attribute assignment method for a depth camera of the present application can, when performing target detection on RGB data, not only detect the target frame and the target category of the detected target by using a deep learning-based method, but also freely define the category attributes of the detected target based on an open source target frame annotation tool to obtain the target detection result (as shown). Figure 4 shown).

[0065] For example, Figure 2 As shown, step S100 of the point cloud attribute assignment method for a depth camera of the present application may include the following steps:

[0066] S110: Detecting the RGB data using a deep learning-based target detection model to obtain the target frame and the target category of the detected target; and

[0067] S120: freely defining the category of the detection target based on an open source target box annotation model to obtain the category attribute of the detection target.

[0068] It is worth noting that the point cloud attribute assignment method for a depth camera of the present application is based on the RGBD data of the depth camera as input, that is, it utilizes the texture information of RGB for target detection and category diversification, and utilizes the spatial information of the point cloud and the recognition results of RGB for point cloud segmentation and attribute assignment. It can well solve the current problem of little and difficult three-dimensional recognition annotation data, and can more freely utilize the existing open source RGB data to realize the customization of categories, and then realize the analog attribute assignment of the point cloud.

[0069] At the same time, the point cloud attribute assignment method for the depth camera of the present application also uses a deep learning target detection method to detect the RGB data, so that the detection accuracy is higher and the speed is faster, and the category definition is richer and more flexible, which helps to adapt to the needs of different scenarios.

[0070] According to the above embodiments of the present application, Figure 3 As shown, the step S200 of the point cloud attribute assignment method for a depth camera may include the following steps:

[0071] S210: Preliminarily segmenting the point cloud data using an RGBD mapping model to obtain an initial point cloud region corresponding to the target frame of the detection target;

[0072] S220: performing ground detection on the initial point cloud area to obtain a ground-removed point cloud area;

[0073] S230: performing dynamic histogram statistics on the ground-removed point cloud area to obtain the number of point clouds in each bin interval; and

[0074] S240: Determine whether the maximum bin interval and the second largest bin interval are adjacent. If so, select two continuous intervals corresponding to the maximum bin interval and the second largest bin interval respectively as the target point cloud area; if not, select a continuous area corresponding to the maximum bin interval as the target point cloud area.

[0075] Preferably, if Figure 3 As shown, the step S200 of the point cloud attribute assignment method for a depth camera further includes the following steps before the step S220:

[0076] S250: performing point cloud preprocessing on the initial point cloud area through connected domain filtering to obtain a denoised point cloud area, so that ground detection is subsequently performed on the denoised point cloud area to obtain the ground-deleted point cloud area.

[0077] For example, taking the case where the depth camera is placed horizontally, after performing target detection on the RGB data to obtain the target detection result, RGBD mapping is first performed according to the target detection result to find the corresponding initial point cloud area; then, a method such as connected domain filtering is used to remove a part of the point cloud noise in the initial point cloud area to obtain the denoised point cloud area; then, a small distance truncation is made in the Z-axis direction perpendicular to the ground to remove the influence of ground noise and obtain the ground-removed point cloud area; then, a dynamic vertical truncation is made on the ground-removed point cloud area according to the positions of all points in the Z-axis direction. A histogram is counted (for example, the distance between the nearest point and the farthest point in the Z-axis direction of the point cloud area to the ground is counted with a bin interval of every 10 centimeters) to obtain the number of points in each bin interval (e.g., the number of point clouds within a distance of 10 centimeters corresponding to the bin interval); finally, based on whether the largest bin interval (i.e., the bin interval with the largest number of points) and the second largest bin interval (i.e., the bin interval with the second largest number of points) are adjacent, one or two consecutive bin intervals are dynamically selected as the final point cloud segmentation result, that is, the target point cloud area is accurately obtained to complete the accurate segmentation of the point cloud data.

[0078] It is worth noting that the point cloud attribute assignment method for a depth camera of the present application can quickly find the corresponding point cloud area and the RGB information corresponding to the point cloud area through the RGBD mapping method, so as to make full use of color information to assist spatial information, making segmentation more accurate. At the same time, the point cloud attribute assignment method for a depth camera also uses ground detection and dynamic histogram statistics to remove the interference of the ground and background in the point cloud area to the greatest extent, so as to accurately locate the target point cloud area. In particular, the point cloud attribute assignment method for a depth camera of the present application is simpler and more feasible than the existing direct use of point cloud information for three-dimensional point cloud object recognition, but it can achieve the same effect, and is also a three-dimensional point cloud object recognition method based on TOF.

[0079] It is worth mentioning that in the step S300 of the point cloud attribute assignment method for the depth camera of the present application, after obtaining the accurate target point cloud area, the target point cloud area can be considered to be the point cloud corresponding to the target detection result, and then the target point cloud area is assigned the corresponding category attribute (such as Figure 6 As shown in Figure 2, the point clouds of the two regions are assigned as potted plants and chairs respectively).

[0080] Schematic system

[0081] Reference to the accompanying drawings Figure 7 As shown in FIG, a point cloud attribute assignment system for a depth camera according to an embodiment of the present invention is illustrated. Specifically, as Figure 7 As shown, the point cloud attribute assignment system 500 for a depth camera includes: a detection module 510, which is communicatively connected to each other, and is used to perform target detection processing on the two-dimensional image data obtained by the depth camera to obtain a target detection result, wherein the target detection result includes a target frame, a target category and a corresponding category attribute of one or more detection targets; a segmentation module 520, which is used to perform point cloud segmentation processing on the point cloud data obtained by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain a target point cloud area; and an assignment module 530, which is used to assign the category attribute of the detected target to the corresponding target point cloud area.

[0082] It is worth noting that the depth camera is used to collect RGBD data, wherein the two-dimensional image data is RGB data.

[0083] In one example of this application, Figure 7 As shown, the detection module 510 includes: a target detection module 511, which is communicatively connected to each other, and is used to detect the RGB data through a target detection model based on deep learning to obtain the target box and the target category of the detected target; and a target labeling module 512, which is used to freely define the category of the detected target through a target box labeling model based on open source to obtain the category attribute of the detected target.

[0084] In one example of this application, Figure 7 As shown, the segmentation module 520 includes: a preliminary segmentation module 521, which is used to perform preliminary segmentation on the point cloud data through the RGBD mapping model to obtain an initial point cloud area corresponding to the target box of the detection target; a ground detection module 522, which is used to perform ground detection on the initial point cloud area to obtain a point cloud area removed from the ground; a histogram statistics module 523, which is used to perform dynamic histogram statistics on the point cloud area removed from the ground to obtain the number of point clouds in each bin interval; and a judgment and selection module 524, which is used to judge whether the maximum bin interval and the second largest bin interval are adjacent. If so, two continuous intervals corresponding to the maximum bin interval and the second largest bin interval are selected as the target point cloud area; if not, a continuous area corresponding to the maximum bin interval is selected as the target point cloud area.

[0085] It is worth mentioning that Figure 7 As shown, the segmentation module 520 further includes a preprocessing module 525, wherein the preprocessing module is used to perform point cloud preprocessing on the initial point cloud area through connected domain filtering to obtain a denoised point cloud area, so that the denoised point cloud area is subsequently subjected to ground detection to obtain the denoised point cloud area.

[0086] Schematic electronic devices

[0087] Below, reference Figure 8 An electronic device according to an embodiment of the present invention is described below. Figure 8 As shown, the electronic device 90 includes one or more processors 91 and a memory 92 .

[0088] The processor 91 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions. In other words, the processor 91 includes one or more physical devices configured to execute instructions. For example, the processor 91 may be configured to execute instructions as a part of the following: one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions may be implemented to perform tasks, implement data types, convert the state of one or more components, implement technical effects, or otherwise obtain desired results.

[0089] The processor 91 may include one or more processors configured to execute software instructions. In addition or in lieu thereof, the processor 91 may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the processor 91 may be single-core or multi-core, and the instructions executed thereon may be configured for serial, parallel and / or distributed processing. The various components of the processor 91 may optionally be distributed across two or more separate devices, which may be remotely located and / or configured to perform collaborative processing. Various aspects of the processor 91 may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0090] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement some or all of the steps in the above-described illustrative method of the present invention, and / or other desired functions.

[0091] In other words, the memory 92 includes one or more physical devices configured to store machine-readable instructions that can be executed by the processor 91 to implement the methods and processes described herein. When implementing these methods and processes, the state of the memory 92 can be changed (e.g., to store different data). The memory 92 can include removable and / or internal devices. The memory 92 can include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray Disc, etc.), semiconductor storage (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic storage (e.g., hard drive, floppy disk drive, tape drive, MRAM, etc.), etc. The memory 92 can include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.

[0092] It will be appreciated that the memory 92 comprises one or more physical devices. However, various aspects of the instructions described herein may alternatively be transmitted via a communication medium (e.g., electromagnetic signals, optical signals, etc.) that is not held by a physical device for a limited period of time. Various aspects of the processor 91 and the memory 92 may be integrated together into one or more hardware logic components. These hardware logic components may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASICs), program and application specific standard products (PSSP / ASSPs), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0093] In one example, if Figure 8 As shown, the electronic device 90 may also include an input device 93 and an output device 94, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). For example, the input device 93 may be, for example, a camera module for collecting image data or video data, etc. As another example, the input device 93 may include or interface with one or more user input devices such as a keyboard, a mouse, a touch screen, or a game controller. In some embodiments, the input device 93 may include or interface with selected natural user input (NUI) components. Such component parts may be integrated or peripheral, and the transduction and / or processing of input actions may be handled on-board or off-board. Example NUI components may include a microphone for language and / or speech recognition; infrared, color, stereoscopic display, and / or depth cameras for machine vision and / or gesture recognition; a head tracker, eye tracker, accelerometer, and / or gyroscope for motion detection and / or intent recognition; and an electric field sensing component for evaluating brain activity and / or body movement; and / or any other suitable sensor.

[0094] The output device 94 can output various information to the outside, including classification results, etc. The output device 94 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0095] Of course, the electronic device 90 may further include the communication device, wherein the communication device may be configured to communicatively couple the electronic device 90 to one or more other computer devices. The communication device may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication device may allow the electronic device 90 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0096] It will be understood that the configurations and / or methods described herein are exemplary in nature, and that these specific embodiments or examples should not be considered restrictive, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions shown and / or described may be performed in the order shown and / or described, in other orders, in parallel, or omitted. Likewise, the order of the above-described processes may be changed.

[0097] Of course, to simplify, Figure 8 Only some of the components related to the present invention in the electronic device 90 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device 90 may further include any other appropriate components according to specific application scenarios.

[0098] It should also be noted that in the apparatus, device and method of the present invention, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present invention.

[0099] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0100] Those skilled in the art will appreciate that the embodiments of the present invention described above and shown in the accompanying drawings are intended to be illustrative only and are not intended to limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and illustrated in the embodiments. Any variations or modifications may be made to the embodiments of the present invention without departing from the principles described.

Claims

1. A method for assigning attributes to a point cloud for a depth camera, characterized in that: Including steps: Performing target detection processing on the two-dimensional image data acquired by the depth camera to obtain a target detection result, wherein the target detection result includes a target frame, a target category, and corresponding category attributes of one or more detected targets; Performing point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain a target point cloud area; as well as Assigning the category attribute of the detected target to the corresponding target point cloud area; The step of performing point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain the target point cloud area includes the following steps: Performing preliminary segmentation on the point cloud data to obtain an initial point cloud area corresponding to the target frame of the detection target; The initial point cloud region is preprocessed by connected domain filtering to obtain a denoised point cloud region; Performing ground detection on the denoised point cloud area to obtain a ground-removed point cloud area; Perform dynamic histogram statistics on the ground-removed point cloud area to obtain the number of point clouds in each bin interval; and Determine whether the maximum bin interval and the second largest bin interval are adjacent. If so, select two continuous intervals corresponding to the maximum bin interval and the second largest bin interval respectively as the target point cloud area; if not, select a continuous area corresponding to the maximum bin interval as the target point cloud area.

2. The point cloud attribute assignment method for a depth camera according to claim 1, wherein: The depth camera is used to collect RGBD data, wherein the two-dimensional image data is RGB data.

3. The point cloud attribute assignment method for a depth camera according to claim 2, wherein: The step of performing target detection processing on the two-dimensional image data acquired by the depth camera to obtain a target detection result, wherein the target detection result includes a target frame, a target category, and corresponding category attributes of one or more detected targets, comprises the steps of: Detecting the RGB data using a deep learning-based target detection model to obtain the target frame and target category of the detected target; and By freely defining the category of the detection target based on the open source target box annotation model, the category attributes of the detection target are obtained.

4. The point cloud attribute assignment method for a depth camera according to any one of claims 1 to 3, wherein: The depth camera is a TOF camera.

5. A point cloud attribute assignment system for a depth camera, characterized in that: including communicatively connected to each other: a detection module, configured to perform target detection processing on the two-dimensional image data acquired by the depth camera to obtain a target detection result, wherein the target detection result includes a target frame, target category, and corresponding category attributes of one or more detected targets; a segmentation module, configured to perform point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera, so as to obtain a target point cloud area; as well as an assignment module, configured to assign the category attribute of the detected target to the corresponding target point cloud area; The segmentation module includes: a preliminary segmentation module communicatively connected to each other, for performing preliminary segmentation on the point cloud data to obtain an initial point cloud area corresponding to the target frame of the detection target; A preprocessing module, wherein the preprocessing module is used to perform point cloud preprocessing on the initial point cloud area through connected domain filtering to obtain a denoised point cloud area; a ground detection module is used to perform ground detection on the denoised point cloud area to obtain a ground-removed point cloud area; a histogram statistics module is used to perform dynamic histogram statistics on the ground-removed point cloud area to obtain the number of point clouds in each bin interval; and a judgment and selection module is used to judge whether the maximum bin interval and the second largest bin interval are adjacent, and if so, select two continuous intervals corresponding to the maximum bin interval and the second largest bin interval respectively as the target point cloud area; if not, select a continuous area corresponding to the maximum bin interval as the target point cloud area.

6. The point cloud attribute assignment system for a depth camera according to claim 5, wherein: The depth camera is used to collect RGBD data, wherein the two-dimensional image data is RGB data.

7. The point cloud attribute assignment system for a depth camera according to claim 6, wherein: The detection module includes: a target detection module communicatively connected to each other, configured to detect the RGB data using a deep learning-based target detection model to obtain the target frame and the target category of the detected target; and a target labeling module for freely defining the category of the detection target based on an open source target box labeling model to obtain the category attribute of the detection target.

8. An electronic device, characterized in that include: at least one processor for executing instructions; and A memory communicatively connected to the at least one processor, wherein the memory has at least one instruction, wherein the instruction is executed by the at least one processor to cause the at least one processor to perform some or all steps of a point cloud attribute assignment method for a depth camera, wherein the point cloud attribute assignment method for a depth camera comprises the steps of: Performing target detection processing on the two-dimensional image data acquired by the depth camera to obtain a target detection result, wherein the target detection result includes a target frame, a target category, and corresponding category attributes of one or more detected targets; performing point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain a target point cloud area; and Assigning the category attribute of the detected target to the corresponding target point cloud area; The step of performing point cloud segmentation processing on the point cloud data acquired by the depth camera based on the target detection result according to the two-dimensional to three-dimensional mapping relationship of the depth camera to obtain the target point cloud area includes the following steps: Performing preliminary segmentation on the point cloud data to obtain an initial point cloud area corresponding to the target frame of the detection target; The initial point cloud region is preprocessed by connected domain filtering to obtain a denoised point cloud region; Performing ground detection on the denoised point cloud area to obtain a ground-removed point cloud area; Perform dynamic histogram statistics on the ground-removed point cloud area to obtain the number of point clouds in each bin interval; and Determine whether the maximum bin interval and the second largest bin interval are adjacent. If so, select two continuous intervals corresponding to the maximum bin interval and the second largest bin interval respectively as the target point cloud area; if not, select a continuous area corresponding to the maximum bin interval as the target point cloud area.

Images