Point cloud indexing method, apparatus, computer readable storage medium, and terminal device

Abstract

The application belongs to the technical field of point cloud processing, and particularly relates to a point cloud indexing method and device, a computer readable storage medium and a terminal device. The method comprises: acquiring a preset RGBD image collected by a depth camera; acquiring camera intrinsic parameters of the depth camera, and converting the RGBD image into ordered point cloud data according to the camera intrinsic parameters; and establishing an index of the ordered point cloud data in a deep learning process according to an arrangement mode of each point cloud element in the ordered point cloud data. Through the application, the one-time indexing establishment process is used to replace the frame-by-frame searching process in the traditional method, the time cost of the most time-consuming searching process in point cloud deep learning is compressed to zero time cost, the training and reasoning efficiency of point cloud deep learning is greatly improved, and the real-time requirements of various application scenarios are met.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of point cloud processing, and particularly relates to a point cloud indexing method and device, a computer readable storage medium and a terminal device. BACKGROUND

[0002] Fast real-time semantic perception of massive point clouds, such as semantic segmentation or classification, is a key technology for intelligent perception in application scenarios such as robot navigation and positioning and autonomous driving. The current deep learning-based semantic understanding of massive point clouds faces great challenges in real-time performance. For example, the extraction of point cloud features based on deep learning methods generally aggregates the neighborhood point set through a k-nearest neighbor search algorithm and extracts local features, which is crucial for improving the performance of point cloud semantic understanding. Although there are many existing acceleration algorithms for k-nearest neighbor search, such as kd-tree, when faced with massive point clouds, the frame-by-frame and point-by-point nearest neighbor search still requires a large amount of time overhead, which greatly reduces the efficiency of semantic segmentation and cannot meet the real-time requirements. SUMMARY

[0003] Therefore, the embodiments of the present application provide a point cloud indexing method, device, computer readable storage medium and terminal device to solve the problem of low processing efficiency of point cloud data in the prior art, which cannot meet the real-time requirements.

[0004] The first aspect of the embodiments of the present application provides a point cloud indexing method, which can include:

[0005] obtaining an RGBD image collected by a preset depth camera;

[0006] obtaining camera intrinsic parameters of the depth camera, and converting the RGBD image into ordered point cloud data according to the camera intrinsic parameters;

[0007] performing a downsampling operation on the ordered point cloud data according to a preset downsampling parameter to obtain downsampling point cloud data corresponding to the ordered point cloud data; and mapping each point cloud element in the downsampling point cloud data to an index of each point cloud element in the ordered point cloud data as a downsampling index applied in a process of extracting global semantic features;

[0008] for any one point cloud element in the ordered point cloud data, determining a sliding window region centered on the point cloud element; mapping the point cloud element to an index of a point cloud element in the sliding window region as a nearest neighbor index of the point cloud element; and traversing each point cloud element in the ordered point cloud data to obtain a nearest neighbor index applied in a process of extracting nearest neighbor points;

[0009] According to a preset downsampling parameter, the ordered point cloud data is downsampled to obtain downsampled point cloud data corresponding to the ordered point cloud data; and indexes of each point cloud element in the ordered point cloud data corresponding to each point cloud element in the downsampled point cloud data are taken as upsampled indexes applied in a process of mapping global information to a local detail region.

[0010] In an implementation form of the first aspect, each point cloud element in the ordered point cloud data corresponds to each pixel in the RGBD image one by one, and an arrangement manner of each point cloud element in the ordered point cloud data is consistent with an arrangement manner of each pixel in the RGBD image.

[0011] In an implementation form of the first aspect, the camera intrinsic parameters include a horizontal focal length, a vertical focal length and a light center position, and the conversion of the RGBD image into the ordered point cloud data according to the camera intrinsic parameters includes:

[0012] For any one pixel in the RGBD image, a point cloud element corresponding to the pixel is calculated according to the following formula:

[0013]

[0014] z=d

[0015] wherein f x is the horizontal focal length of the depth camera, f y is the vertical focal length of the depth camera, (c x ,c y ) is the light center position of the depth camera, (u,v) is the position of the pixel, d is the distance corresponding to the pixel, and (x,y,z) is the point cloud element corresponding to the pixel.

[0016] Each pixel in the RGBD image is traversed to obtain the ordered point cloud data.

[0017] The second aspect of the embodiment of the application provides a point cloud index device, which can include:

[0018] An image acquisition module is configured to acquire an RGBD image collected by a preset depth camera.

[0019] A point cloud data conversion module is configured to acquire camera intrinsic parameters of the depth camera and convert the RGBD image into ordered point cloud data according to the camera intrinsic parameters.

[0020] The index establishing module is configured to: perform down-sampling on the ordered point cloud data according to a preset down-sampling parameter to obtain down-sampled point cloud data corresponding to the ordered point cloud data; correspond an index of each point cloud element in the down-sampled point cloud data to an index of each point cloud element in the ordered point cloud data as a down-sampling index applied in a process of extracting global semantic features; for any one point cloud element in the ordered point cloud data, determine a sliding window region centered on the point cloud element; correspond an index of the point cloud element to an index of a point cloud element in the sliding window region as a neighbor index of the point cloud element; traverse each point cloud element in the ordered point cloud data to obtain a neighbor index applied in a process of extracting neighbor points; perform down-sampling on the ordered point cloud data according to a preset down-sampling parameter to obtain down-sampled point cloud data corresponding to the ordered point cloud data; and correspond an index of each point cloud element in the ordered point cloud data to an index of each point cloud element in the down-sampled point cloud data as an up-sampling index applied in a process of mapping global information to a local detail region.

[0021] In a specific implementation manner of the second aspect, each point cloud element in the ordered point cloud data corresponds to each pixel in the RGBD image one by one, and an arrangement manner of each point cloud element in the ordered point cloud data is consistent with an arrangement manner of each pixel in the RGBD image.

[0022] In a specific implementation manner of the second aspect, the index establishing module can include:

[0023] The down-sampling index establishing unit is configured to: perform down-sampling on the ordered point cloud data according to a preset down-sampling parameter to obtain down-sampled point cloud data corresponding to the ordered point cloud data; and correspond an index of each point cloud element in the down-sampled point cloud data to an index of each point cloud element in the ordered point cloud data as a down-sampling index of the ordered point cloud data.

[0024] In a specific implementation manner of the second aspect, the index establishing module can include:

[0025] The neighbor index establishing unit is configured to: for any one point cloud element in the ordered point cloud data, determine a sliding window region centered on the point cloud element; correspond an index of the point cloud element to an index of a point cloud element in the sliding window region as a neighbor index of the point cloud element; and traverse each point cloud element in the ordered point cloud data to obtain a neighbor index of the ordered point cloud data.

[0026] In a specific implementation manner of the second aspect, the index establishing module can include:

[0027] The up-sampling index establishing unit is configured to perform a down-sampling operation on the ordered point cloud data according to a preset down-sampling parameter to obtain down-sampled point cloud data corresponding to the ordered point cloud data; and an index of each point cloud element in the ordered point cloud data corresponding to each point cloud element in the down-sampled point cloud data is taken as an up-sampling index of the ordered point cloud data.

[0028] In a specific implementation manner of the second aspect, the camera intrinsic parameters can include a horizontal focal length, a vertical focal length and a light center position, and the point cloud data conversion module is specifically configured to calculate a point cloud element corresponding to any one pixel in the RGBD image according to the following formula:

[0029]

[0030] z=d

[0031] wherein f x is the horizontal focal length of the depth camera, f y is the vertical focal length of the depth camera, (c x ,c y ) is the light center position of the depth camera, (u,v) is the position of the pixel, d is the distance corresponding to the pixel, and (x,y,z) is the point cloud element corresponding to the pixel.

[0032] The third aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of any one of the point cloud index methods.

[0033] The fourth aspect of the embodiment of the present application provides a terminal device, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of any one of the point cloud index methods when executing the computer program.

[0034] The fifth aspect of the embodiment of the present application provides a computer program product, which, when executed on a terminal device, causes the terminal device to perform the steps of any one of the point cloud index methods.

[0035] The beneficial effects of the embodiments of the present application compared with the prior art are: the embodiments of the present application obtain a preset RGBD image collected by a depth camera; obtain camera intrinsic parameters of the depth camera, and convert the RGBD image into ordered point cloud data according to the camera intrinsic parameters; and establish an index of the ordered point cloud data in a deep learning process according to an arrangement mode of each point cloud element in the ordered point cloud data. Through the embodiments of the present application, the one-time index establishment process is used to replace the frame-by-frame search process in the traditional method, the time cost of the most time-consuming search process in point cloud deep learning is compressed to zero time cost, the training and inference efficiency of point cloud deep learning is greatly improved, and the real-time requirements of various application scenarios are met. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0037] Figure 1 An embodiment flowchart of a point cloud indexing method in the embodiments of the present application;

[0038] Figure 2 A schematic diagram of down-sampling index;

[0039] Figure 3 A schematic diagram of near-neighbor index;

[0040] Figure 4 A schematic diagram of up-sampling index;

[0041] Figure 5 An embodiment structure diagram of a point cloud indexing device in the embodiments of the present application;

[0042] Figure 6 A schematic block diagram of a terminal device in the embodiments of the present application. DETAILED DESCRIPTION

[0043] In order to make the purposes, characteristics and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the embodiments described below are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0044] It should be understood that the word "comprising" when used in this specification and annexed claims, specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0046] It should further be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items, and that the term "at least one of' denotes one, or a plurality of, or any combination of the listed items.

[0047] As used in this specification and the appended claims, the term "if' can be construed to mean "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be construed to mean "once it is determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]," depending on the context.

[0048] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0049] Please refer to Figure 1 One embodiment of a point cloud indexing method in the embodiments of the present application can include:

[0050] In step S101, an RGBD image collected by a preset depth camera is acquired.

[0051] In the embodiments of the present application, the RGBD image can include two images: one is an R(Red, red) G(Green, green) B(Blue, blue) three-channel color image, and the other is a depth image. The depth image is similar to a gray-scale image, except that each pixel value is the distance from the sensor to the object. Usually, the RGB image and the depth image are registered, so there is a one-to-one correspondence between the pixel points.

[0052] Optionally, after the RGBD image is acquired, a series of pre-processing can be performed thereon, which can include but is not limited to image scaling, depth value scaling, data type conversion, etc.

[0053] In the embodiments of the present application, the size of the final obtained RGBD image is denoted as W*H, wherein W and H are generally set as an integer power of 2 or an integer multiple of a high power of 2, such as an integer multiple of 64.

[0054] In step S102, camera intrinsic parameters of the depth camera are obtained, and the RGBD image is converted into ordered point cloud data according to the camera intrinsic parameters.

[0055] Each point cloud element in the ordered point cloud data corresponds to each pixel in the RGBD image one by one, and the arrangement mode of each point cloud element in the ordered point cloud data is consistent with the arrangement mode of each pixel in the RGBD image.

[0056] The camera intrinsic parameters can include a horizontal focal length, a vertical focal length and a light center position. Taking any one pixel in the RGBD image as an example, the point cloud element corresponding to the pixel can be calculated according to the following formula:

[0057]

[0058] z=d

[0059] wherein f x is the horizontal focal length of the depth camera, f y is the vertical focal length of the depth camera, (c x ,c y ) is the light center position of the depth camera, (u,v) is the position of the pixel, d is the distance corresponding to the pixel, and (x,y,z) is the point cloud element corresponding to the pixel.

[0060] The point cloud elements corresponding to each pixel in the RGBD image (a total of W*H pixels) are calculated in the above manner, and these point cloud elements (a total of W*H point cloud elements) constitute the ordered point cloud data.

[0061] It should be noted that, due to the limitation of the depth sensor or the special material object that cannot obtain depth information, the pixel value corresponding to the depth image is an invalid value, and the position of the invalid value is randomly changed. Therefore, if the point cloud element corresponding to the random invalid value is removed, the order of the point cloud will be destroyed. Therefore, in the process of generating the point cloud, the last point cloud element is used to replace the point cloud element corresponding to the invalid value, which not only ensures the order of the point cloud, but also ensures the effectiveness and authenticity of the point cloud element. In particular, if the first pixel value of the depth image is an invalid value, the corresponding point cloud element is set as (0, 0, 0).

[0062] Step S103, establishing an index of the ordered point cloud data in a deep learning process according to an arrangement mode of each point cloud element in the ordered point cloud data.

[0063] The index of the ordered point cloud data in the deep learning process can include but is not limited to a down-sampling index, a near-neighbor index and an up-sampling index.

[0064] The point cloud semantic understanding deep learning network generally extracts global features in a coding-decoding structure. The coding process is generally a down-sampling process, aiming to extract global semantic features, and the point cloud data needs to be down-sampled at this time. In the embodiments of the present application, the down-sampling index of the ordered point cloud data can be established by using the uniform sampling method of the point cloud data.

[0065] Figure 2 The figure is a schematic diagram of the down-sampling index. When the down-sampling index is established, the ordered point cloud data can be first down-sampled according to a preset down-sampling parameter to obtain down-sampled point cloud data corresponding to the ordered point cloud data.

[0066] The down-sampling parameter can be set according to actual conditions, which is not specifically limited in the embodiments of the present application. For example, if the down-sampling parameter is 2, there are H / 2*W / 2 point cloud elements in the down-sampled point cloud data.

[0067] Then, the index of each point cloud element in the down-sampled point cloud data corresponding to each point cloud element in the ordered point cloud data can be taken as the down-sampling index of the ordered point cloud data.

[0068] Since the near-neighbor relationship of the point cloud elements can represent the near-neighbor relationship between the three-dimensional points, the near-neighbor points of the point cloud can be quickly extracted by using the constraint between the point cloud elements. That is, a fixed-size sliding window is preset, and the point cloud elements contained in the sliding window are regarded as the near-neighbor points of the point cloud at each position of the sliding window.

[0069] Figure 3 The figure is a schematic diagram of the near-neighbor index. When the near-neighbor index is established, for any point cloud element in the ordered point cloud data, a sliding window region centered on the point cloud element can be first determined, and the index of the point cloud element in the sliding window region corresponding to the point cloud element can be taken as the near-neighbor index of the point cloud element.

[0070] The size of the sliding window region can be set according to actual conditions, which is not specifically limited in the embodiments of the present application. For example, the size of the sliding window region can be set to 5*5, and 25 near-neighbor points can be obtained by the sliding window method, in which the middle point is the point cloud element itself.

[0071] It should be noted that the sliding window area at the edge of the ordered point cloud data needs to be ensured not to slide out of the range of the ordered point cloud data, so as to ensure that all indexes are valid indexes.

[0072] According to the above manner, each point cloud element in the ordered point cloud data is traversed, and the corresponding neighbor index of each point cloud element is obtained, thereby forming the neighbor index of the ordered point cloud data.

[0073] The decoder is commonly used in a two-dimensional image deep learning network, and the structure can effectively map global information to a local detail area. However, in a point cloud deep learning network, the structure is difficult to implement, mainly because of the unordered nature of the point cloud, and it is difficult to apply a nearest neighbor or interpolation method for upsampling. Therefore, an embodiment of the present application utilizes the ordered nature of the point cloud to establish the upsampling index of the ordered point cloud data.

[0074] Figure 4 The above is a schematic diagram of the upsampling index. When the upsampling index is established, the ordered point cloud data can be first down-sampled according to a preset down-sampling parameter to obtain down-sampled point cloud data corresponding to the ordered point cloud data, and then the index of each point cloud element in the ordered point cloud data corresponding to each point cloud element in the down-sampled point cloud data is taken as the upsampling index of the ordered point cloud data.

[0075] In the same task, assuming that the resolution of the input RGBD image remains unchanged, the index used in the point cloud deep learning network remains unchanged, and therefore, the down-sampling index, the neighbor index, and the upsampling index can be extracted in advance according to the resolution of the input image. Only one-time index generation is required, and the index can be repeatedly used in the subsequent deep learning process.

[0076] In summary, an embodiment of the present application obtains a preset RGBD image collected by a depth camera, obtains camera intrinsic parameters of the depth camera, and converts the RGBD image into ordered point cloud data according to the camera intrinsic parameters. According to the arrangement mode of each point cloud element in the ordered point cloud data, the index of the ordered point cloud data in the deep learning process is established. Through the embodiment of the present application, the one-time index establishment process is used to replace the frame-by-frame search process in the traditional method, the time-consuming search process in the point cloud deep learning is compressed to zero time cost, the training and inference efficiency of the point cloud deep learning is greatly improved, and the real-time requirement of various application scenarios is met.

[0077] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0078] A point cloud indexing method, Figure 5 An embodiment structure diagram of a point cloud indexing device is shown.

[0079] In an embodiment, a point cloud indexing device can include:

[0080] An image acquisition module 501 is configured to acquire an RGBD image collected by a preset depth camera.

[0081] A point cloud data conversion module 502 is configured to acquire camera intrinsic parameters of the depth camera and convert the RGBD image into ordered point cloud data according to the camera intrinsic parameters.

[0082] An index establishing module 503 is configured to establish an index of the ordered point cloud data in a deep learning process according to an arrangement manner of each point cloud element in the ordered point cloud data.

[0083] In a specific implementation manner of the embodiment, each point cloud element in the ordered point cloud data corresponds to each pixel in the RGBD image one-to-one, and the arrangement manner of each point cloud element in the ordered point cloud data is consistent with the arrangement manner of each pixel in the RGBD image.

[0084] In a specific implementation manner of the embodiment, the index of the ordered point cloud data in the deep learning process can include a down-sampling index, and the index establishing module can include:

[0085] A down-sampling index establishing unit is configured to perform a down-sampling operation on the ordered point cloud data according to a preset down-sampling parameter to obtain down-sampling point cloud data corresponding to the ordered point cloud data, and correspond each point cloud element in the down-sampling point cloud data to an index of each point cloud element in the ordered point cloud data as a down-sampling index of the ordered point cloud data.

[0086] In a specific implementation manner of the embodiment, the index of the ordered point cloud data in the deep learning process can include a neighbor index, and the index establishing module can include:

[0087] A neighbor index establishing unit is configured to determine a sliding window region centered on each point cloud element in the ordered point cloud data, correspond the point cloud element to an index of a point cloud element in the sliding window region as a neighbor index of the point cloud element, and traverse each point cloud element in the ordered point cloud data to obtain a neighbor index of the ordered point cloud data.

[0088] In one specific implementation of this application embodiment, the index of the ordered point cloud data in the deep learning process may include an upsampling index, and the index building module may include:

[0089] An upsampling index establishment unit is used to perform downsampling operations on the ordered point cloud data according to preset downsampling parameters to obtain downsampled point cloud data corresponding to the ordered point cloud data; and to use the index of each point cloud element in the ordered point cloud data to correspond to each point cloud element in the downsampled point cloud data as the upsampling index of the ordered point cloud data.

[0090] In one specific implementation of this application embodiment, the camera intrinsic parameters may include: horizontal focal length, vertical focal length, and optical center position. The point cloud data conversion module is specifically used to calculate the point cloud element corresponding to any pixel in the RGBD image according to the following formula:

[0091]

[0092] z = d

[0093] Among them, f x f is the horizontal focal length of the depth camera. y Let (c) be the vertical focal length of the depth camera. x ,c y Let (u,v) be the optical center position of the depth camera, (u,v) be the position of the pixel, d be the distance corresponding to the pixel, and (x,y,z) be the point cloud element corresponding to the pixel; traverse each pixel in the RGBD image to obtain the ordered point cloud data.

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0095] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0096] Figure 6 A schematic block diagram of a terminal device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0097] like Figure 6 As shown, the terminal device 6 in this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps in the various point cloud indexing method embodiments described above, for example...Figure 1 The processor 60 implements the functions of the above-mentioned modules / units in the apparatus embodiments when executing the computer program 62. For example, the processor 60 implements the functions of the modules 501-503 shown in FIG. 5 when executing the computer program 62. Figure 5 The processor 60 implements the functions of the above-mentioned modules / units in the apparatus embodiments when executing the computer program 62. For example, the processor 60 implements the functions of the modules 501-503 shown in FIG. 5 when executing the computer program 62.

[0098] For example, the computer program 62 can be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 62 in the terminal device 6.

[0099] The terminal device 6 can be a mobile phone, a tablet computer, a desktop computer, a notebook computer, a palm computer, a robot, and other computing devices. Those skilled in the art can understand that the terminal device 6 is not limited to the examples shown in the figures, and can include more or fewer components than those shown, or combine certain components, or different components, for example, the terminal device 6 can also include an input / output device, a network access device, a bus, and the like. Figure 6 The terminal device 6 shown in the figures is merely an example and does not constitute a limitation on the terminal device 6, and can include more or fewer components than those shown, or combine certain components, or different components, for example, the terminal device 6 can also include an input / output device, a network access device, a bus, and the like.

[0100] The processor 60 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0101] The memory 61 can be an internal storage unit of the terminal device 6, for example, a hard disk or a memory of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 6. Further, the memory 61 can also include both the internal storage unit and the external storage device of the terminal device 6. The memory 61 is used to store the computer program and other programs and data required by the terminal device 6. The memory 61 can also be used to temporarily store data that has been output or is to be output.

[0102] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0103] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0104] Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0105] In the embodiments of the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other manners. For example, the embodiments of the apparatus / terminal device described above are merely schematic, and the division of the modules or units is merely logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0106] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0107] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0108] The integrated module / unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable storage medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer readable storage medium can include or exclude contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable storage medium does not include electrical carrier signals and telecommunication signals.

[0109] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A point cloud indexing method, characterized in that, The method comprises the following steps: acquiring an RGBD image collected by a preset depth camera; acquiring camera intrinsic parameters of the depth camera, and converting the RGBD image into ordered point cloud data according to the camera intrinsic parameters; establishing an index of the ordered point cloud data in a deep learning process according to an arrangement mode of each point cloud element in the ordered point cloud data; the index of the ordered point cloud data in the deep learning process comprises a down-sampling index applied in a process of extracting global semantic features, a neighbor index applied in a process of extracting neighbor points, and an up-sampling index applied in a process of mapping global information to a local detail region; wherein, the ordered point cloud data is down-sampled according to a preset down-sampling parameter, to obtain down-sampled point cloud data corresponding to the ordered point cloud data; and an index of each point cloud element in the down-sampled point cloud data corresponding to each point cloud element in the ordered point cloud data is taken as the down-sampling index applied in the process of extracting global semantic features; for any one point cloud element in the ordered point cloud data, a sliding window region centered on the point cloud element is determined; an index of the point cloud element corresponding to a point cloud element in the sliding window region is taken as the neighbor index of the point cloud element; each point cloud element in the ordered point cloud data is traversed to obtain the neighbor index applied in the process of extracting neighbor points; the ordered point cloud data is down-sampled according to a preset down-sampling parameter, to obtain down-sampled point cloud data corresponding to the ordered point cloud data; and an index of each point cloud element in the ordered point cloud data corresponding to each point cloud element in the down-sampled point cloud data is taken as the up-sampling index applied in the process of mapping global information to a local detail region.

2. The point cloud indexing method of claim 1, wherein, each point cloud element in the ordered point cloud data corresponds to each pixel in the RGBD image one by one, and an arrangement mode of each point cloud element in the ordered point cloud data is consistent with an arrangement mode of each pixel in the RGBD image.

3. The point cloud indexing method of any one of claims 1-2, wherein, the camera intrinsic parameters comprise a horizontal focal length, a vertical focal length and a light center position, and the conversion of the RGBD image into the ordered point cloud data according to the camera intrinsic parameters comprises: for any one pixel in the RGBD image, a point cloud element corresponding to the pixel is calculated according to the following formula: z = d where f x is the horizontal focal length of the depth camera, f y is the vertical focal length of the depth camera, (c x , c y ) is the principal point of the depth camera, (u, v) is the position of the pixel, d is the distance corresponding to the pixel, and (x, y, z) is the point cloud element corresponding to the pixel. each pixel in the RGBD image is traversed to obtain the ordered point cloud data.

4. A point cloud indexing apparatus, characterized by comprising: The method comprises the following steps: an image acquisition module is configured to acquire an RGBD image collected by a preset depth camera; a point cloud data conversion module is configured to acquire camera intrinsic parameters of the depth camera, and convert the RGBD image into ordered point cloud data according to the camera intrinsic parameters; an index establishment module is configured to establish an index of the ordered point cloud data in a deep learning process according to an arrangement mode of each point cloud element in the ordered point cloud data; The indexes of the ordered point cloud data in the deep learning process include: a down-sampling index applied in a process of extracting global semantic features, a neighbor index applied in a process of extracting neighbor points, and an up-sampling index applied in a process of mapping global information to a local detail area; wherein, the ordered point cloud data is down-sampled according to a preset down-sampling parameter to obtain down-sampled point cloud data corresponding to the ordered point cloud data; indexes of each point cloud element in the down-sampled point cloud data corresponding to each point cloud element in the ordered point cloud data are taken as the down-sampling index applied in the process of extracting global semantic features; for any one point cloud element in the ordered point cloud data, a sliding window region centered on the point cloud element is determined; indexes of the point cloud element corresponding to point cloud elements in the sliding window region are taken as the neighbor index of the point cloud element; each point cloud element in the ordered point cloud data is traversed to obtain the neighbor index applied in the process of extracting neighbor points; the ordered point cloud data is down-sampled according to a preset down-sampling parameter to obtain down-sampled point cloud data corresponding to the ordered point cloud data; indexes of each point cloud element in the ordered point cloud data corresponding to each point cloud element in the down-sampled point cloud data are taken as the up-sampling index applied in the process of mapping global information to a local detail area.

5. The point cloud indexing apparatus of claim 4, wherein, The camera intrinsic parameters include a horizontal focal length, a vertical focal length and a light center position, and the point cloud data conversion module is specifically configured to calculate, for any one pixel in the RGBD image, a point cloud element corresponding to the pixel according to the following formula: z=d wherein f x is the horizontal focal length of the depth camera, f y is the vertical focal length of the depth camera, (c x , c y ) is the optical center position of the depth camera, (u, v) is the position of the pixel, d is the distance corresponding to the pixel, and (x, y, z) is the point cloud element corresponding to the pixel. The ordered point cloud data is obtained by traversing each pixel in the RGBD image.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. The computer program, when executed by a processor, implements the steps of the point cloud index method according to any one of claims 1 to 3.

7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program, when executed by a processor, implements the steps of the point cloud index method according to any one of claims 1 to 3.

Images