A point cloud generation method, device and automatic driving device

Abstract

Embodiments of the present application provide a point cloud generation method and device and an autonomous driving device, and relate to the technical field of 3D vision. The point cloud generation method comprises: obtaining point cloud data of an initial point cloud for a target object; inputting the point cloud data of the initial point cloud into a pre-trained point cloud generation network to obtain an output result of the point cloud generation network; and determining a target point cloud for the target object by using the output result. It can be seen that the present application can feed back high-level feature data learned by a high-level network to a low-level network to improve the semantic expression ability of low-level feature data, that is, the high-level semantic information learned by the high-level network can be transmitted to the low-level network to participate in the learning of low-level feature data, thereby assisting the learning of low-level feature data, improving the semantic expression ability of low-level feature data, and ultimately improving the quality of generated point clouds, that is, the accuracy and integrity of generated point clouds can be improved.

Description

Technical Field

[0001] This invention relates to the field of 3D vision technology, and in particular to a point cloud generation method, apparatus, and autonomous driving device. Background Technology

[0002] Point clouds can be applied in fields such as 3D modeling, 3D recognition, and autonomous navigation. However, due to factors such as target object self-occlusion, sensor resolution, object material, and environment, the initial point cloud acquired by the acquisition device for the target object inevitably has problems such as missing data, low resolution, and noise. Therefore, after obtaining the initial point cloud of the target object, it is usually necessary to perform point cloud generation processing based on the initial point cloud. The purpose is to perform point cloud repair, point cloud denoising, and point cloud upsampling, thereby ensuring the accuracy and completeness of the point cloud of the target object.

[0003] In related technologies, during the point cloud generation process, point cloud data flows from the lower-level network of the point cloud generation network to the higher-level network. Among them, the sub-network that is closer to the final generated point cloud is the higher-level network, and the sub-network that is farther away from the final generated point cloud is the lower-level network.

[0004] However, in related technologies, point cloud generation network information only flows from low-level networks to high-level networks. Due to the lack of effective guidance from high-level semantic information, low-level networks can only focus on learning low-level semantic information, resulting in insufficient semantic expression ability of low-level feature data, which ultimately affects the quality of generated point clouds, that is, the accuracy and completeness of generated point clouds are not high. Summary of the Invention

[0005] The purpose of this invention is to provide a point cloud generation method, apparatus, and autonomous driving device to improve the accuracy and completeness of point clouds. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of the present invention provide a point cloud generation method, applied to an electronic device, the method comprising:

[0007] Obtain the point cloud data of the initial point cloud for the target object;

[0008] The point cloud data of the initial point cloud is input into a pre-trained point cloud generation network to obtain the output result of the point cloud generation network. The point cloud generation network includes at least one feedback sub-network with feedback functionality. Each feedback sub-network is used to learn point cloud features using the acquired input content to obtain output content. When a predetermined output content feedback condition is met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network continues to learn point cloud features based on the feedback content. If a next sub-network exists, when a predetermined output content transmission condition is met, the current output content is input into the next sub-network. The designated sub-network corresponding to any feedback sub-network includes: the feedback sub-network and / or the sub-network preceding the feedback sub-network.

[0009] Using the output results, the target point cloud for the target object is determined.

[0010] Optionally, when the predetermined output content feedback conditions are met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the designated sub-network can continue to learn point cloud features based on the feedback content, including:

[0011] When the number of times the output content has been fed back is less than a preset threshold, the current output content is used as the feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network can continue to learn point cloud features based on the feedback content.

[0012] The step of inputting the current output content into the next sub-network when the predetermined output content transmission conditions are met includes:

[0013] If the current output content has not been passed to the next subnetwork in the feedback subnetwork, the current output content is input into the next subnetwork.

[0014] Optionally, when the predetermined output content feedback conditions are met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the designated sub-network can continue to learn point cloud features based on the feedback content, including:

[0015] When the number of times the output content has been fed back is less than a preset threshold, the current output content is used as the feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network can continue to learn point cloud features based on the feedback content.

[0016] The step of inputting the current output content into the next sub-network when the predetermined output content transmission conditions are met includes:

[0017] When the number of feedback attempts for the output content is less than a preset threshold, the current output content is input into the next sub-network.

[0018] Optionally, the output results generated by the point cloud generation network during training include the output results corresponding to each time step; the time step is a processing stage divided according to time sequence.

[0019] The output result corresponding to each time step is the output result generated by each sub-network in the point cloud generation network through deep point cloud feature learning using the data content corresponding to that time step; wherein, for any specified sub-network in each sub-network, the data content corresponding to that time step includes at least the latest output content of the content transmission end corresponding to that specified sub-network, and the feedback content of the feedback sub-network corresponding to that specified sub-network from the previous time step; the data content corresponding to other sub-networks in each sub-network includes at least the latest output content of the corresponding content transmission end; wherein, the content transmission end corresponding to any sub-network in the point cloud generation network is located before that sub-network and is used to transmit content to that sub-network;

[0020] Determining the target point cloud for the target object using the output results includes:

[0021] The target point cloud for the target object is determined by using the output result corresponding to the latest time step in each time step.

[0022] Optionally, before inputting the point cloud data of the initial point cloud into a pre-trained point cloud generation network to obtain the output of the point cloud generation network, the method further includes:

[0023] The point cloud generation network is obtained as the target time step during the operation of the electronic device. The target time step is any one of the various time steps and is set based on the device type of the electronic device and / or the available computing power of the electronic device.

[0024] The value of the preset number threshold is updated to the order value of the target time step in each time step minus 1.

[0025] Optionally, the method of setting the time step to be utilized based on the device type of the electronic device includes:

[0026] If the electronic device is a device of the first type of device, the time step to be utilized is determined to be the earliest time step among all time steps;

[0027] If the electronic device is a device of the second type, the time step to be utilized is determined to be any time step other than the earliest time step and the latest time step among all time steps;

[0028] If the electronic device is a third type of device, the time step to be utilized is determined to be the latest time step among all time steps;

[0029] Among them, the computing power of the first type of equipment, the second type of equipment, and the third type of equipment gradually increases.

[0030] Optionally, after the point cloud generation network is trained, the parameter values ​​of the network parameters of the point cloud generation network are the same at different time steps.

[0031] Optionally, the designated subnetwork corresponding to any feedback subnetwork is specifically used to perform fusion processing on the currently received feedback content and the latest output content of the corresponding content transmission end each time feedback content is received, and to use the fused content obtained by the fusion processing to perform point cloud feature learning; wherein, the content transmission end corresponding to any subnetwork in the point cloud generation network is located before the subnetwork and is used to transmit content to the subnetwork.

[0032] Optionally, the fusion processing of the currently received feedback content and the latest output content from the corresponding content transmission end includes:

[0033] For each point in the latest output content of the corresponding content transmission terminal, among the points of the currently received feedback content, at least one neighboring point is determined. Using the feature data of at least one neighboring point and the feature data of the point, a specified feature fusion process is performed to obtain the fused data of the point.

[0034] Optionally, the fusion processing of the currently received feedback content and the latest output content from the corresponding content transmission end includes:

[0035] For each point in the latest output content of the corresponding content transmission end, calculate the attention weight of each point in the currently received feedback content for that point, and obtain the relevance weight matrix corresponding to that point;

[0036] For each point in the latest output content of the corresponding content transmission terminal, the feature data of each point in the currently received feedback content are weighted according to the relevance weight matrix corresponding to that point to obtain the feature data to be used for that point. The feature data of that point is then fused with the feature data to be used for that point to obtain the fused data of that point.

[0037] Optionally, for each point in the latest output content of the corresponding content transmission end, among the points in the currently received feedback content, at least one neighboring point corresponding to that point is determined, including:

[0038] For each point in the latest output content of the corresponding content transmission end, the K nearest neighbor algorithm (KNN) is used to determine the K nearest points to the current point from each point in the currently received feedback content, and at least one neighboring point corresponding to the current point is obtained.

[0039] or,

[0040] For each point in the latest output content of the corresponding content transmission end, the attention weight of that point with each point in the currently received feedback content is determined using the relevance weight matrix corresponding to that point; based on the determined attention weight, at least one neighboring point is determined from each point in the currently received feedback content; wherein, the relevance weight matrix corresponding to that point is used to characterize the attention weight of that point with each point in the currently received feedback content.

[0041] Optionally, the point cloud generation network is obtained by training the network based on distance loss or adversarial loss; wherein the distance loss includes chamfered distance (CD) loss or plain distance (EMD) loss.

[0042] Secondly, embodiments of the present invention provide a point cloud generation device, applied to an electronic device, the device comprising:

[0043] The acquisition module is used to acquire point cloud data of the initial point cloud for the target object;

[0044] An input module is used to input the point cloud data of the initial point cloud into a pre-trained point cloud generation network to obtain the output result of the point cloud generation network; wherein, the point cloud generation network includes at least one feedback sub-network with feedback function, each feedback sub-network is used to learn point cloud features using the acquired input content to obtain output content, and when a predetermined output content feedback condition is met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network continues to learn point cloud features based on the feedback content, and if there is a next sub-network, when a predetermined output content transmission condition is met, the current output content is input into the next sub-network; wherein, the designated sub-network corresponding to any feedback sub-network includes: the feedback sub-network and / or the sub-network located before the feedback sub-network;

[0045] The determination module is used to determine the target point cloud for the target object using the output results.

[0046] Thirdly, embodiments of the present invention provide an autonomous driving device, including: a data acquisition module and a processing module;

[0047] The acquisition module is used to generate an initial point cloud for the target object;

[0048] The processing module is used to generate a target point cloud for the target object using point cloud data of the initial point cloud for the target object, according to any of the point cloud generation methods described above.

[0049] Optionally, the processing module is further configured to, upon receiving an obstacle avoidance task or navigation task for the target object, execute the obstacle avoidance task or navigation task for the target object based on the target point cloud of the target object.

[0050] Optionally, the acquisition module includes a depth camera;

[0051] The depth camera is used to acquire images of the target object and to construct an initial point cloud for the target object using the acquired images.

[0052] The autonomous driving device also includes a driving component that drives the autonomous driving device to move based on the target point cloud of the target object.

[0053] Optionally, the acquisition module includes a lidar;

[0054] The lidar is used to collect radar data from the target object and to construct an initial point cloud for the target object using the collected radar data.

[0055] The autonomous driving device also includes a driving component that drives the autonomous driving device to move based on the target point cloud of the target object.

[0056] The solution provided in this invention obtains point cloud data of an initial point cloud for a target object, inputs the point cloud data into a trained point cloud generation network, and obtains the output result of the point cloud generation network. The point cloud generation network includes at least one feedback subnetwork. The feedback subnetwork performs point cloud feature learning on the point cloud data to obtain output content. When a predetermined output content feedback condition is met, the current output content is used as feedback content and fed back to the corresponding designated subnetwork so that the designated subnetwork continues to use the feedback content as input content for point cloud feature learning. Furthermore, if a next subnetwork exists, when a predetermined output content transmission condition is met, the current output content is input into the next subnetwork. Then, the target point cloud for the target object is determined using the output result. It can be seen that this solution can feed back high-level feature data learned by high-level networks to low-level networks to improve the semantic expressive power of low-level feature data. That is, it can transmit high-level semantic information learned by high-level networks to low-level networks to participate in the learning of low-level feature data, thereby assisting the low-level feature data in learning, improving the semantic expressive power of low-level feature data, and ultimately improving the quality of the generated point cloud, i.e., improving the accuracy and completeness of the generated point cloud. In addition, compared with related technologies that improve the accuracy and completeness of point clouds by stacking a large number of network modules, this solution improves the accuracy and completeness of point clouds through a feedback mechanism, which enables a lightweight point cloud generation network, thereby ultimately improving the overall network performance.

[0057] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0059] Figure 1 This is a flowchart illustrating a point cloud generation method provided in an embodiment of the present invention;

[0060] Figure 2 A flowchart illustrating another point cloud generation method provided in an embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram illustrating the working principle of a point cloud generation network provided in an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram illustrating the working principle of another point cloud generation network provided in an embodiment of the present invention;

[0063] Figure 5 A schematic diagram illustrating the generation effect of a point cloud generation network provided in an embodiment of the present invention;

[0064] Figure 6 This is a schematic diagram of the structure of a point cloud generation device provided in an embodiment of the present invention;

[0065] Figure 7 This is a schematic diagram of the structure of an autonomous driving device provided in an embodiment of the present invention;

[0066] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.

[0068] Below, we will first introduce the technical terms involved in the embodiments of this invention:

[0069] Point cloud completion / point cloud repair: a technical method for reconstructing and restoring the missing parts of the original point cloud, where the original point cloud can also be called the initial point cloud.

[0070] Point cloud denoising: a method that identifies and filters noise points in the original point cloud to generate a new, clean point cloud without noise. Noise points can also be called wild points.

[0071] Point cloud upsampling: Upsampling the sparse original point cloud to generate a fine-grained representation of the target object.

[0072] The low-level and high-level networks of a point cloud generation network: During point cloud generation, the characteristics of point cloud data flow from the low-level network to the high-level network. The low-level network consists of layers relatively far from the output result, while the high-level network consists of layers relatively close to the output result. Generally, the low-level network focuses more on extracting local semantics from the original input data, while the high-level network focuses more on the global target and extracting abstract semantic information.

[0073] To improve the accuracy and completeness of point clouds, embodiments of the present invention provide a point cloud generation method, apparatus, and autonomous driving device.

[0074] The following section first introduces a point cloud generation method provided by the embodiments of the invention.

[0075] The point cloud generation method provided in this embodiment of the invention can be applied to electronic devices. In specific applications, the electronic device can be a terminal device or a server, and this embodiment of the invention does not limit the specific form of the electronic device.

[0076] Specifically, the execution entity of this point cloud generation method can be a point cloud generation device. For example, when the point cloud generation method is applied to a terminal device, the point cloud generation device can be a client running on the terminal device for point cloud generation. For example, when the point cloud generation method is applied to a server, the point cloud generation device can be a computer program running on the server, which can be used for point cloud generation.

[0077] The point cloud generation method provided in this embodiment of the invention may include:

[0078] Obtain the point cloud data of the initial point cloud for the target object;

[0079] The point cloud data of the initial point cloud is input into a pre-trained point cloud generation network to obtain the output result of the point cloud generation network. The point cloud generation network includes at least one feedback sub-network with feedback functionality. Each feedback sub-network is used to learn point cloud features using the acquired input content to obtain output content. When a predetermined output content feedback condition is met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network continues to learn point cloud features based on the feedback content. If a next sub-network exists, when a predetermined output content transmission condition is met, the current output content is input into the next sub-network. The designated sub-network corresponding to any feedback sub-network includes: the feedback sub-network and / or the sub-network preceding the feedback sub-network.

[0080] Using the output results, the target point cloud for the target object is determined.

[0081] The solution provided in this invention obtains point cloud data of an initial point cloud for a target object, inputs the point cloud data into a trained point cloud generation network, and obtains the output result of the point cloud generation network. The point cloud generation network includes at least one feedback subnetwork. The feedback subnetwork performs point cloud feature learning on the point cloud data to obtain output content. When a predetermined output content feedback condition is met, the current output content is used as feedback content and fed back to the corresponding designated subnetwork so that the designated subnetwork continues to use the feedback content as input content for point cloud feature learning. Furthermore, if a next subnetwork exists, when a predetermined output content transmission condition is met, the current output content is input into the next subnetwork. Then, the target point cloud for the target object is determined using the output result. It can be seen that this solution can feed back high-level feature data learned by high-level networks to low-level networks to improve the semantic expressive power of low-level feature data. That is, it can transmit high-level semantic information learned by high-level networks to low-level networks to participate in the learning of low-level feature data, thereby assisting the low-level feature data in learning, improving the semantic expressive power of low-level feature data, and ultimately improving the quality of the generated point cloud, i.e., improving the accuracy and completeness of the generated point cloud. In addition, compared with related technologies that improve the accuracy and completeness of point clouds by stacking a large number of network modules, this solution improves the accuracy and completeness of point clouds through a feedback mechanism, which enables a lightweight point cloud generation network, thereby ultimately improving the overall network performance.

[0082] The following description, in conjunction with the accompanying drawings, introduces a point cloud generation method provided by an embodiment of the present invention.

[0083] like Figure 1 As shown in the figure, a point cloud generation method provided by an embodiment of the present invention may include the following steps:

[0084] S101, Obtain point cloud data for the initial point cloud of the target object;

[0085] The target object mentioned above can be any object that requires point cloud generation. The initial point cloud can be obtained by a data acquisition device that collects data from the target object. This acquisition device can be a LiDAR, depth camera, or similar device. For ease of understanding, let's take a robotic vacuum cleaner equipped with a depth camera as an example to illustrate the generation of the target object and the initial point cloud: The depth camera needs to collect data on obstacles around the robotic vacuum cleaner for obstacle avoidance. Therefore, the obstacles can be the target object. The depth map captured by the depth camera can be converted into an initial point cloud for the obstacles. It should be noted that the point cloud data of the initial point cloud can be specified information for each point in the initial point cloud, such as coordinate information, normal direction, and color information. Furthermore, the target object can be changed according to the specific usage scenario; this embodiment of the invention does not limit this.

[0086] Understandably, initial point clouds are typically acquired by acquisition devices. Due to factors such as target object self-occlusion, sensor resolution (i.e., the acquisition device's resolution), object material, and environment, initial point clouds generally inevitably suffer from issues like missing points, low resolution, and noise. For example, a face acquired by a 3D acquisition device may exhibit missing points and low resolution due to insufficient sensor resolution. Therefore, initial point clouds directly acquired by acquisition devices are generally unusable for downstream vision tasks, or they may negatively impact the processing performance of those tasks. Thus, point cloud generation methods are needed to generate point clouds with high accuracy and completeness to improve the performance of downstream vision tasks. The purpose of point cloud generation is to perform point cloud repair, denoising, and upsampling on the initial point cloud for the target object, thereby ensuring the accuracy and completeness of the target object's point cloud.

[0087] Furthermore, it should be emphasized that the embodiments of the present invention do not limit the process of acquiring the initial point cloud for the target object; any method that can acquire the initial point cloud can be applied to the embodiments of the present invention. Moreover, the embodiments of the present invention do not limit the specific representational form of the point cloud data of the initial point cloud.

[0088] S102, the point cloud data of the initial point cloud is input into a pre-trained point cloud generation network to obtain the output result of the point cloud generation network; wherein, the point cloud generation network includes at least one feedback sub-network with feedback function, each feedback sub-network is used to learn point cloud features using the acquired input content to obtain output content, and when a predetermined output content feedback condition is met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network continues to learn point cloud features based on the feedback content, and if there is a next sub-network, when a predetermined output content transmission condition is met, the current output content is input into the next sub-network; wherein, the designated sub-network corresponding to any feedback sub-network includes: the feedback sub-network and / or the sub-network located before the feedback sub-network;

[0089] When inputting the initial point cloud data into the point cloud generation network, if the initial point cloud data meets the input requirements of the point cloud generation network, it can be directly input into the network. If the initial point cloud data does not meet the input requirements, it can be pre-processed according to the network's requirements before being input. For example, the electronic device can perform coordinate normalization on the input point cloud data, such as scaling and translating the coordinates to between -1 and 1, to make it meet the network's input requirements. Correspondingly, after obtaining the output result, the output result can be inversely normalized to restore the data, such as restoring the coordinates to their original coordinate system values.

[0090] The aforementioned point cloud generation network includes at least one feedback subnetwork with feedback functionality. This feedback mechanism enhances the semantic representation of low-level feature data, improving the learning performance of the low-level network and thus increasing the accuracy and completeness of the point cloud. The low-level feature data is learned by the low-level network, and the high-level feature data is learned by the high-level network. It should be noted that the point cloud generation network may include only one feedback subnetwork; for example, if multiple feedback subnetworks exist, they can be connected in series. Of course, in addition to including at least one feedback subnetwork, the point cloud generation network may also include a feedforward subnetwork. For example, if both feedback and feedforward subnetworks exist in the point cloud generation network, they can also be connected in series. It is understood that the connection method of the subnetworks is not limited to series connection. The feedforward subnetwork does not have a feedback function. After learning point cloud features from the input data, the input content is directly passed to the next network. Furthermore, the feedforward subnetwork can consist of one or more point cloud learning modules, such as one or more modules selected from MLPs (Multi-Layer Perceptions), DGGCN (Dynamic Graph CNN), and Point Transformer feature extractors. This invention does not specifically limit the composition of this network. The output of the feedforward subnetwork is point cloud features. For example, these point cloud features can be feature data about each point in the point cloud, or they can be high-dimensional features about the point cloud. The high-dimensional features can be in vector form, but are not limited to this. It is understood that if the point cloud features are feature data about each point in the point cloud, then the points involved in the point cloud features can be the points of the point cloud represented by the input content, i.e., the points of the input point cloud, or they can be the points of a new point cloud formed based on the input content; both are reasonable. In this embodiment of the invention, the feedforward subnetwork may or may not be omitted, and the invention does not impose specific limitations on this.

[0091] The feedback subnetwork can also be composed of one or more point cloud learning modules as described above. It should be noted that compared to the feedforward subnetwork, the feedback subnetwork adds a feedback module. The function of the feedback module is to fuse high-level feature data with low-level feature data, improving the learning ability of the low-level network and thus improving the overall network performance. Furthermore, it should be understood that the above-mentioned subnetworks are only for ease of description. Content feedback can occur between subnetworks, and also within a single subnetwork, to transfer high-level feature data from the network to the low-level network. In specific applications, for point cloud generation networks, provided that the definitions of feedback and feedforward subnetworks are met, the point cloud generation network can be divided into multiple parts, defined as multiple subnetworks, according to actual needs, to achieve high-level to low-level content feedback. This invention does not limit the specific division method. The output content of each feedback subnetwork can be point cloud features. For a description of point cloud features, please refer to the description of point cloud features in the feedforward subnetwork above; it will not be repeated here. For example, feedback subnetwork 0 and feedback subnetwork 1 are connected in series. Feedback subnetwork 1 is closer to the output result than feedback subnetwork 0. Feedback subnetwork 0 is located in a lower-level network, while feedback subnetwork 1 is located in a higher-level network. Feedback subnetwork 1 feeds back the feature data of each output point to the corresponding feedback subnetwork 0, so that feedback subnetwork 0 can continue to learn point cloud features using the received feedback content as input. This invention does not limit the output content of the feedback subnetwork. Furthermore, it is understood that the number of feedback iterations of the feedback subnetwork can be determined based on computing power, which can be understood as the processing power of a computer chip. If the computing power is sufficient, the number of feedback iterations can be increased to achieve better results. If the computing power is weak, the number of feedback iterations can be reduced to achieve moderate algorithm performance. If the computing power is severely insufficient, feedback can be omitted.

[0092] To facilitate understanding, the following is an example of how the point cloud generation network works, with reference to the accompanying drawings. Figure 3 An exemplary schematic diagram illustrating the working principle of a point cloud generation network provided in an embodiment of the present invention is given.

[0093] like Figure 3As shown, the point cloud generation network includes one feedforward subnetwork and n feedback subnetworks, i.e., feedforward subnetwork and feedback subnetworks 0 to n. The arrows pointing from the output of each feedback subnetwork back to its input in the diagram represent feedback connections. During point cloud generation, the point cloud is input to the feedforward subnetwork. After point cloud feature learning, the feedforward subnetwork passes its output to the n interconnected feedback subnetworks. Each of the n feedback subnetworks uses the acquired input to learn point cloud features. When a predetermined feedback condition is met, it uses the current output as feedback to continue learning point cloud features based on the feedback. If a next subnetwork exists, it inputs the current output to that subnetwork when a predetermined transmission condition is met. If no next feedback subnetwork exists, the result is output. It should be noted that... Figure 3 This is merely an example and should not be construed as limiting the embodiments of the present invention.

[0094] In addition, for example, the designated subnetwork corresponding to any of the above feedback subnetworks is specifically used to perform fusion processing on the currently received feedback content and the latest output content of the corresponding content transmission end each time feedback content is received, and to use the fused content obtained by the fusion processing to learn point cloud features; wherein, the content transmission end corresponding to any of the above point cloud generation networks is located before the subnetwork and is used to transmit content to the subnetwork.

[0095] It should be noted that if the feedback content received by the sub-network corresponds one-to-one with the latest output content of the corresponding content transmission end, feature splicing can be performed directly.

[0096] In this context, the latest output content of the corresponding content transmission terminal can be the latest output content of the preceding subnetwork of the specified subnetwork. If the specified subnetwork is the first subnetwork of the entire point cloud generation network, then the latest output content of the corresponding content transmission terminal is the input content of the point cloud generation network. In this case, the corresponding content transmission terminal is the input terminal of the point cloud generation network. It should be noted that a low-level network can receive feedback from multiple high-level networks; conversely, high-level feature data can also be fed back to multiple low-level networks. Furthermore, for example, if the output content is high-dimensional features of the point cloud, the currently received feedback content and the latest output content of the corresponding content transmission terminal can be weighted to complete the fusion process, but this is not limited to this. If the output content is feature data of each point in the point cloud, there are multiple ways to perform fusion processing. For clarity and layout, the following describes the specific implementation of fusing the currently received feedback content and the latest output content of the corresponding content transmission terminal through other embodiments. It is understood that in other embodiments, the specified subnetwork corresponding to the above-mentioned feedback subnetwork can also be used to directly learn point cloud features from the feedback content each time feedback content is received. For example, the point cloud generation network has a feedback subnetwork 0 and a feedback subnetwork 1. The feedback subnetwork 0 and the feedback subnetwork 1 are connected in series. The feedback subnetwork 1 is closer to the output result than the feedback subnetwork 0. Therefore, the feedback subnetwork 0 is a low-level network and the feedback subnetwork 1 is a high-level network. The feedback subnetwork 1 provides feedback to the feedback subnetwork 0. The feedback subnetwork 0 can use the feedback content to improve the semantic expression ability of its output features.

[0097] There are multiple ways to implement the predetermined output content feedback conditions and the predetermined output content transmission conditions. Optionally, in a first implementation, when the predetermined output content feedback conditions are met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the designated sub-network can continue to learn point cloud features based on the feedback content, including:

[0098] When the number of times the output content has been fed back is less than a preset threshold, the current output content is used as the feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network can continue to learn point cloud features based on the feedback content.

[0099] The step of inputting the current output content into the next sub-network when the predetermined output content transmission conditions are met includes:

[0100] If the current output content has not been passed to the next subnetwork in the feedback subnetwork, the current output content is input into the next subnetwork.

[0101] As can be seen, in this first implementation, the output content feedback condition is that the number of times the output content has been fed back is less than a preset threshold; while the output content transmission condition is that the feedback sub-network has not transmitted the current output content to the next sub-network.

[0102] In the second implementation, the step of using the current output content as feedback content when the predetermined output content feedback condition is met, and feeding it back to the corresponding designated sub-network so that the designated sub-network can continue to learn point cloud features based on the feedback content, includes:

[0103] When the number of times the output content has been fed back is less than a preset threshold, the current output content is used as the feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network can continue to learn point cloud features based on the feedback content.

[0104] Accordingly, the step of inputting the current output content into the next sub-network when the predetermined output content transmission conditions are met includes:

[0105] When the number of feedback attempts for the output content is less than a preset threshold, the current output content is input into the next sub-network.

[0106] As can be seen, in this second implementation, the output content feedback condition is that the number of times the output content has been fed back is less than a preset threshold; and the output content transmission condition is that the number of times the output content has been fed back is less than a preset threshold.

[0107] It should be noted that, for example, the above two predetermined output content feedback conditions also include the case where the current computing power of the electronic device meets the predetermined computing power sufficiency condition, that is, when the current computing power of the electronic device allows it.

[0108] Regarding the first implementation method: In this method, each time the feedback subnetwork generates an output, it is passed to the next subnetwork. This allows the output of the feedback subnetwork to be applied to the learning process of the next subnetwork, thereby further improving the learning effect of the low-level and high-level networks. That is, it enhances the semantic representation ability of the low-level and high-level feature data, and further improves the accuracy and completeness of the point cloud.

[0109] It should be noted that, based on the first implementation method described above, the output results generated by the point cloud generation network during training include the output results corresponding to each time step; the time step is a processing stage divided according to time sequence; the output result corresponding to each time step is the output result generated by each sub-network in the point cloud generation network using the data content corresponding to that time step to perform deep point cloud feature learning; wherein, for any specified sub-network in each sub-network, the data content corresponding to that time step at least includes the latest output content of the corresponding content transmission end of the specified sub-network, and the feedback content of the corresponding feedback sub-network of the specified sub-network in the previous time step, and the data content corresponding to other sub-networks in each sub-network at least includes the latest output content of the corresponding content transmission end; wherein, the content transmission end corresponding to any sub-network in the point cloud generation network is located before that sub-network and is used to transmit content to that sub-network.

[0110] It is understandable that, for the first implementation method mentioned above, the predetermined output content feedback condition can be considered as the current time step being less than the total number of preset time steps during network training, that is, the current time step being less than the total number of time steps unfolded by the point cloud generation network.

[0111] The data content corresponding to this time step includes at least the latest output content of the corresponding content transmission end of the specified sub-network, and the feedback content of the feedback sub-network corresponding to the specified sub-network in the previous time step. It should be noted that this is not limited to the feedback content of the feedback sub-network corresponding to the specified sub-network in the previous time step; it can also include the feedback content of the time step preceding the time step, and even the feedback content of all time steps above this time step. This invention does not limit this. It can be understood that the latest output content of the corresponding content transmission end is, that is, the output content of the previous-level sub-network in this time step.

[0112] Furthermore, the preset threshold value is an initial default value. This initial default value represents the maximum number of feedback responses to the output content that the current point cloud generation network can achieve when expanding across all time steps. During the training process of the point cloud generation network, the preset threshold value is used as the initial default value. This ensures that the output results for all time steps are obtained during model training, which is the preset threshold value plus the generation results for one time step. At this point, the point cloud generation network expands across all time steps. It should be noted that after training, the network parameters of the point cloud generation network are identical across different time steps; that is, the network parameters are shared across different time steps.

[0113] For example, the output at each time step is specifically the output generated by each sub-network in the point cloud generation network using the data content corresponding to that time step and performing deep point cloud feature learning according to the network parameters shared by the point cloud generation network at each time step. That is, the network parameters of the point cloud generation network are the same when performing feature learning at different time steps. Here, network parameters can be understood as the configuration information of the point cloud generation network, such as convolutional kernels, etc. It should be noted that the network parameters can also be not shared; in this case, the network parameters of the point cloud generation network are different at each time step.

[0114] Since the network parameters of the point cloud generation network are shared across different time steps, this reduces the number of network parameters and the difficulty of network optimization, enabling a lightweight point cloud generation network and ultimately improving overall network performance. Without a feedback mechanism, more network modules need to be stacked to achieve better performance, while this solution achieves the same performance with fewer model parameters than existing technologies. Therefore, it is more suitable for applications in mobile devices such as smart locks and smartphones where model size is a constraint.

[0115] The point cloud generation network outputs the results of each time step based on the time-step expansion. For example, if the user sets three time steps, the output of the point cloud generation network will include the output results for all three time steps. It should be noted that the number of time-step expansions is the same as the number of outputs from each feedback subnetwork. For example, the data content at the first time step is the initial output content of each feedback subnetwork.

[0116] To facilitate understanding the working principle of point cloud generation networks, the following explanation, in conjunction with accompanying diagrams, addresses the method of dividing time into various time steps. Figure 4 As shown, the working principle of another point cloud generation network provided in this embodiment of the invention is as follows:

[0117] The point cloud generation network consists of one feedforward subnetwork and n feedback subnetworks. The entire point cloud generation network is expanded in time steps. At time step t0, the point cloud is input to the feedforward subnetwork. After learning the point cloud features, the feedforward subnetwork outputs its content to the feedback subnetwork 0. The feedback subnetwork 0, after learning the point cloud features, outputs its content f. 0,t0 The input is fed into feedback subnetwork 1, which learns point cloud features and outputs content f. 1,t0 The input is fed into subsequent feedback subnetworks until it is fed into feedback subnetwork n. Feedback subnetwork n learns point cloud features and outputs f. n,t0The result is output at time step t0; at time step t1, the feedback subnetwork 0 receives not only the input from the feedforward subnetwork, but also the feedback content f from the subnetwork 0 at time step t0. 0,t0 The feedback subnetwork 0 learns point cloud features to obtain the output content f. 0,t1 The output content f 0,t1 Feedback is sent to subsequent time steps, and the output content f is output. 0,t1 The input is fed into feedback subnetwork 1, which simultaneously receives the feedback content f at time step t0. 1,t0 The content f output by the feedback subnetwork 0 0,t1 As input, point cloud features are learned to obtain the output content f of feedback subnetwork 1. 1,t1 The output content f 1,t1 Feedback is sent to subsequent time steps, and the output content f is output. 1,t1 The input is fed into subsequent feedback subnetworks until it reaches feedback subnetwork n. Feedback subnetwork n learns point cloud features and outputs content f. n，t1 Feedback is fed into the feedback subnetwork n at subsequent time steps, outputting f. n,t1 As the output of time step t1, repeat the above steps until the feedback subnetwork n at time step tm outputs f. n,tm The output is the time step tm.

[0118] It is understandable that the feedback subnetwork 0 at each time step can receive input from the feedforward subnetwork. Alternatively, when the time step is greater than 0, it can receive intermediate results from historical time steps of each subnetwork as input.

[0119] Regarding the second implementation method:

[0120] In this implementation, if the number of times the output content has been fed back reaches a preset threshold, then only the next sub-network is input, and no feedback is given; if the number of times the output content has been fed back is less than the preset threshold, then only feedback is given, and no input is given to the next sub-network. This invention does not limit the preset threshold. It is understood that during the feedback process, the designated sub-network does not necessarily need to fuse the currently received feedback content with the latest output content from the corresponding content transmission end; it can also directly use the received feedback content as input content to perform point cloud feature learning.

[0121] For example, the point cloud generation network has a feedback subnetwork 0 and a feedback subnetwork 1, which are connected in series. Feedback subnetwork 1 is closer to the output result than feedback subnetwork 0, so feedback subnetwork 0 is a lower-level network and feedback subnetwork 1 is a higher-level network. Feedback subnetwork 1 provides feedback to feedback subnetwork 0, with a preset threshold of 5 times. When the number of feedbacks is less than 5, it cannot input to the next subnetwork and can only continue to provide feedback. When the number of feedbacks reaches 5, it cannot provide feedback and can only input the output content to the next subnetwork.

[0122] As can be seen, the above scheme can effectively guarantee the number of feedbacks. If the preset number of feedbacks is not reached, the input to the next sub-network cannot be made. This can ensure the learning effect of the current feedback sub-network, thereby improving the semantic expression ability of low-level feature data and thus improving the accuracy and completeness of the generated point cloud.

[0123] Furthermore, it is understandable that the point cloud generation network in this scheme needs to be pre-trained before use. The point cloud generation network is obtained by training the network using distance loss or adversarial loss. The training process of the point cloud generation network is roughly the same as that in related technologies. For example, during training, a reconstruction loss can be constructed, that is, calculating the distance between the output and the ground truth, such as Chamfer Distance (CD) or Earth Mover's Distance (EMD). Adversarial loss can also be used to increase the realism of the generated results. Optionally, if this scheme can output intermediate point clouds in each sub-network, then loss constraints can also be applied to these intermediate point clouds during training, and loss constraints can also be applied to the output results at each time step. This can make the overall generation effect more realistic. The loss constraint can be the difference between the ground truth and the trained output. For example, feedback subnetwork 0 and feedback subnetwork 1 are connected in series. Feedback subnetwork 1 is closer to the output result than feedback subnetwork 0. Therefore, feedback subnetwork 0 is a low-level network and feedback subnetwork 1 is a high-level network. Feedback subnetwork 1 feeds back to feedback subnetwork 0. The difference between the output result and the true value is used as a loss constraint. In subsequent training, the output result of feedback subnetwork 0 needs to be given a loss constraint before it is input to feedback subnetwork 1. And so on, each output result needs to be given a loss constraint.

[0124] S103, using the output results, determine the target point cloud for the target object.

[0125] The output of this point cloud generation network can be specified information for each point. This specified information can then be used to generate a target point cloud for the target object. Of course, if normalization or other operations have been performed on the input of the point cloud generation network, then inverse normalization or other operations are also needed on the output to obtain the target point cloud for the target object.

[0126] It should be noted that if the feedback subnetwork can output point clouds, then the last subnetwork of the point cloud generation network can be the feedback subnetwork, that is, the output content can be the specified information of each point; of course, if the output content of the feedback subnetwork is the feature content used to characterize the specified information of each point, that is, the output content of the feedback subnetwork is point cloud features, then the last network of the point cloud generation network can also be used to convert the output content of the previous subnetwork into the specified information of each point.

[0127] Furthermore, if the output results generated by the point cloud generation network during training include the output results corresponding to each time step, then determining the target point cloud for the target object using the output results may include: determining the target point cloud for the target object using the output results corresponding to the latest time step among each time step.

[0128] Furthermore, the aforementioned target point cloud can better perform downstream visual tasks. For example, by directly representing the repaired point cloud as identification information in the map, compared with a map constructed using the initial point cloud as identification information, the map constructed using this solution has significant advantages in visualization effect and the completeness of map identification information. Another application is obstacle avoidance. By repairing the invisible areas of the target, the robot vacuum cleaner can more accurately predict impassable areas and achieve more efficient obstacle avoidance.

[0129] The solution provided in this invention obtains point cloud data of an initial point cloud for a target object, inputs the point cloud data into a trained point cloud generation network, and obtains the output result of the point cloud generation network. The point cloud generation network includes at least one feedback subnetwork. The feedback subnetwork performs point cloud feature learning on the point cloud data to obtain output content. When a predetermined output content feedback condition is met, the current output content is used as feedback content and fed back to the corresponding designated subnetwork so that the designated subnetwork continues to use the feedback content as input content for point cloud feature learning. Furthermore, if a next subnetwork exists, when a predetermined output content transmission condition is met, the current output content is input into the next subnetwork. Then, the target point cloud for the target object is determined using the output result. It can be seen that this solution can feed back high-level feature data learned by high-level networks to low-level networks to improve the semantic expressive power of low-level feature data. That is, it can transmit high-level semantic information learned by high-level networks to low-level networks to participate in the learning of low-level feature data, thereby assisting the low-level feature data in learning, improving the semantic expressive power of low-level feature data, and ultimately improving the quality of the generated point cloud, i.e., improving the accuracy and completeness of the generated point cloud. In addition, compared with related technologies that improve the accuracy and completeness of point clouds by stacking a large number of network modules, this solution improves the accuracy and completeness of point clouds through a feedback mechanism, which enables a lightweight point cloud generation network, thereby ultimately improving the overall network performance.

[0130] The following describes, in conjunction with another embodiment, a specific implementation method for fusing the currently received feedback content with the latest output content from the corresponding content transmission end.

[0131] To facilitate understanding, the differences between image tasks and point cloud tasks in the fusion process will be introduced first:

[0132] For example, in image tasks, the feedback content can be a feedback feature map. Since the feature map is regularly and orderly arranged, when the resolution of the feedback feature map is inconsistent with that of the feature map of the lower layer network, operations such as convolution, interpolation, and pooling can be used to align the resolution of the feature maps, and then concat can be used to fuse the two features.

[0133] For example, in point cloud tasks, the feedback content can also be feature data. Since point clouds are irregular and unordered, in most cases, the resolution of the feature data is inconsistent with the resolution of the lower-level network, and there is no mapping relationship. Downsampling the feedback features, i.e., reducing the resolution of the feedback features, cannot align them with the resolution of the lower-level network. Feature fusion techniques require a strict semantic mapping relationship between two point cloud features. That is, the feature data of the point cloud in the lower-level network, through MLPs, EdgeConv (edge ​​convolutional networks), upsampling, etc., generates one or more corresponding high-level semantic features, thus establishing a mapping relationship between the two point cloud features. In this case, the feature data can be fused and downsampled using the mapping relationship, and then the concat operation can be used for final feedback fusion. For point cloud generation tasks, this feature mapping relationship usually does not exist, and image fusion strategies cannot be directly adopted. If such a mapping relationship exists between features in certain layers of the network, it greatly limits the scope and effectiveness of feedback operations in point cloud generation tasks.

[0134] Based on the above description of the differences, the currently received feedback content and the latest output content of the corresponding content transmission end include: feature data of each point. Specifically, the fusion processing of the currently received feedback content and the latest output content of the corresponding content transmission end can include method A1 and method A2:

[0135] Method A1: For each point in the latest output content of the corresponding content transmission end, among the points of the currently received feedback content, determine at least one neighboring point corresponding to that point, and use the feature data of at least one neighboring point and the feature data of that point to perform specified feature fusion processing to obtain the fused data of that point.

[0136] For example, feedback subnetwork 1 and feedback subnetwork 2 are connected in series. Feedback subnetwork 2 is closer to the output result than feedback subnetwork 1. Therefore, feedback subnetwork 1 is a low-level network and feedback subnetwork 2 is a high-level network. Feedback subnetwork 2 feeds back to feedback subnetwork 1. For each point in the latest output point cloud of the corresponding content transmission end, based on the specified information of each point fed back by feedback subnetwork 2 received by feedback subnetwork 1, for each point in the current subnetwork, at least one nearest neighbor point in the feedback point cloud is determined. Using the feature data of at least one neighbor point and the feature data of the point, specified feature fusion processing is performed to obtain the fused data of the point.

[0137] The method for determining the neighboring points of the point in method A1 can include either method A11 or method A12:

[0138] Method A11: For each point in the latest output content of the corresponding content transmission end, the K nearest neighbor algorithm (KNN) is used to determine the K nearest points from each point in the currently received feedback content, and at least one neighboring point corresponding to the point is obtained.

[0139] Method A12: For each point in the latest output content of the corresponding content transmission end, the attention weight between the point and each point in the currently received feedback content is determined using the relevance weight matrix corresponding to the point; based on the determined attention weight, at least one neighboring point is determined from each point in the currently received feedback content; wherein, the relevance weight matrix corresponding to the point is used to characterize the attention weight between the point and each point in the currently received feedback content.

[0140] To facilitate understanding of the method for determining neighborhood points described above, the following explanation is provided in conjunction with the accompanying diagram, as follows: Figure 5 As shown:

[0141] The point cloud PA collected by the device is input into the point cloud generation network to obtain the target point cloud PB with high accuracy and completeness.

[0142] It's important to note that before determining neighboring points, cross-point cloud feature connections need to be established between each point in point cloud PA and each point in point cloud PB. KNN (K-Nearest Neighbors) can be used to construct these connections. Points in the point clouds need to find their nearest neighbors using KNN. For example, point cloud PA uses KNN to find the features of the K most similar points in point cloud PB. KNN can use point cloud coordinates to calculate distances, or it can use other information, such as features learned from point cloud features, or features learned from point cloud features along with their coordinates.

[0143] Besides using KNN to build connections, a global attention mechanism can also be used to calculate the relevance weight matrix between two point clouds when determining neighboring points. Specifically, the attention weight between a point in point cloud PA and each point in point cloud PB is calculated; a higher weight indicates a higher relevance. Then, using the relevance weight matrix, the attention weight between each point and each point in the currently received feedback content is determined. Finally, using the attention weights of each point with each other, the neighboring points of each point are determined. For each point to be determined, the neighboring points have a larger attention weight relative to other points in the currently received feedback content, meaning a higher relevance. The attention weights in the relevance weight matrix corresponding to each point can be learned by a pre-trained neural network, or they can be trained and optimized simultaneously with the point cloud generation task. This embodiment of the invention does not limit the method of generating attention weights. The pre-trained neural network can be trained using sample data and ground truth. The sample data can be the feature data of the target point in the first point cloud and the feature data of the neighboring points of the target point in the second sample point cloud. The ground truth is the attention weights of the neighboring points of the target point for that target point.

[0144] As can be seen, the above scheme can find the neighborhood points corresponding to each point in the point cloud. The above method can fuse the feature data of each point in the point cloud with the feature data of its corresponding neighborhood points, thereby improving the learning effect of the low-level network, that is, improving the semantic expression ability of the low-level feature data, and thus improving the accuracy and completeness of the generated point cloud.

[0145] The process of using feature data from at least one neighboring point and the feature data of the current point to perform a specified feature fusion process to obtain the fused data of the current point may include steps A13-A14:

[0146] Step A13: Calculate the feature difference between the feature data of the point and the feature data of each neighboring point to obtain the edge features corresponding to each neighboring point.

[0147] For example, the feature data of the current point is subtracted from the feature data of the neighboring points to obtain the edge features corresponding to each neighboring point.

[0148] Step A14: Using the edge features corresponding to each neighboring point, determine the fusion content of the point according to the first fusion method or the second fusion method;

[0149] For example, the fusion content of a point is determined by using the edge features corresponding to each neighboring point obtained, according to the first fusion method.

[0150] The first fusion method includes: for each neighboring point, performing feature learning on the edge features corresponding to the neighboring point and the feature data of the point to obtain the first feature data corresponding to the neighboring point; and calculating the average or taking the maximum value of the first feature data corresponding to each neighboring point to obtain the feature data of the point.

[0151] Specifically, the edge features corresponding to a neighboring point and the feature data of that point can be learned using MLPs to obtain the first feature data corresponding to that neighboring point. It can be understood that the first feature data is obtained by fusing the data features of that point with the edge data of its neighbors and then learning through MLPs. For example, for each neighboring point, the edge features corresponding to that neighboring point and the feature data of that point are used to learn features through MLPs to obtain the first feature data corresponding to that neighboring point. The average of the first feature data corresponding to each neighboring point is then calculated to obtain the feature data of that point.

[0152] The second fusion method includes: determining the attention weight of each neighboring point for that point; weighting the edge features corresponding to each neighboring point according to the attention weight of each neighboring point to obtain the second feature data; and fusing the second feature data with the feature data of that point.

[0153] It should be noted that the fusion method can be one or more of the following: concat, transformer, etc. Any fusion method can be applied to the embodiments of the present invention, and the present invention does not limit it.

[0154] The edge features obtained by subtracting the feature data of a point from the feature data of its neighbors can be learned using MLPs to obtain the attention weight of each neighboring point for that point.

[0155] Method A2: For each point in the latest output content of the corresponding content transmission end, calculate the attention weight of each point in the currently received feedback content for that point, and obtain the relevance weight matrix corresponding to that point;

[0156] For each point in the latest output content of the corresponding content transmission terminal, the feature data of each point in the currently received feedback content are weighted according to the relevance weight matrix corresponding to that point to obtain the feature data to be used for that point. The feature data of that point is then fused with the feature data to be used for that point to obtain the fused data of that point.

[0157] The relevance weight matrix represents the attention weights of each point in the received feedback content to that point. This method is suitable for situations using a global attention mechanism. For example, feedback subnetwork 0 and feedback subnetwork 1 are concatenated. Feedback subnetwork 1 is closer to the output result than feedback subnetwork 0, therefore feedback subnetwork 0 is a lower-level network, and feedback subnetwork 1 is a higher-level network. Feedback subnetwork 1 feeds back to feedback subnetwork 0. For each point in the latest output content of the feedforward, the attention weights of each point in the feedback content received by feedback subnetwork 0 to that point in the output content are calculated, resulting in the relevance weight matrix corresponding to that point.

[0158] For example, feedback subnetwork 0, feedback subnetwork 1, and feedback subnetwork 2 are connected in series. Feedback subnetwork 0 is located in a lower layer network, and feedback subnetwork 2 is closer to the output than feedback subnetwork 1. Feedback subnetwork 1 and feedback subnetwork 2 provide feedback to feedback subnetwork 0. For each point in the latest output point cloud of the corresponding content transmission end, the feature data of each point in the point cloud fed back by feedback subnetwork 1 and feedback subnetwork 2 are weighted according to the relevance weight matrix corresponding to that point to obtain the data to be used for that point. The feature data of that point is then concat-fused with the feature data to be used for that point to obtain the fused data of that point.

[0159] As can be seen, the fusion method in this embodiment has good adaptability to the representation form of each output content in the point cloud generation network, which can conveniently and quickly obtain effective fused content, and thus apply it to the point cloud generation process.

[0160] like Figure 2 As shown, another point cloud generation method provided in this embodiment of the invention may include the following steps:

[0161] S201, Obtain point cloud data for the initial point cloud of the target object;

[0162] It should be noted that S201 is the same as S101 in the above embodiments, and will not be described again here.

[0163] S202, obtain the time step to be utilized by the point cloud generation network during the operation of the electronic device, and use it as the target time step; wherein, the time step to be utilized is any one of the various time steps, and the time step to be utilized is set based on the device type of the electronic device and / or the available computing power of the electronic device;

[0164] The electronic device can be a mobile phone, PC, or server, etc., and this invention does not limit this. By determining the target time step, the point cloud generation network can be flexibly applied to different types of electronic devices with varying computing power, provided that the network is trained only once. This ensures both the quality and efficiency of point cloud generation. For example, a point cloud generation network can run for five time steps during training. However, since the network runs on a mobile device with insufficient computing power, only one time step is determined as the target time step.

[0165] It should be noted that the target time step can be set based on the type of electronic device and / or the available computing power of the electronic device. It is understood that the target time can be set based solely on the type of electronic device, or based on the available computing power of the electronic device, or based on both. This invention does not limit this.

[0166] Among them, the methods for setting the time step to be utilized based on the type of electronic device include methods B1-B3:

[0167] Method B1: If the electronic device is a device of the first device type, the time step to be used is determined to be the earliest time step among all time steps;

[0168] Method B2: If the electronic device is a device of the second type, the time step to be used is determined to be any time step other than the earliest time step and the latest time step among all time steps;

[0169] Method B3: If the electronic device is a third type of device, the time step to be utilized is determined to be the latest time step among all time steps;

[0170] Among them, the computing power of the first type of equipment, the second type of equipment, and the third type of equipment gradually increases.

[0171] It is understood that the specific implementation of the device of the first type can be a mobile phone or a microprocessor with weak processing power, etc.; the specific implementation of the device of the second type can be a PC; and the specific implementation of the device of the third type can be a server. This invention does not make any specific limitations on these.

[0172] In embodiments of this invention, when computing power is sufficient, multiple time steps can be used for expansion to achieve better results; when computing power is weak, the number of time steps can be reduced to achieve moderate algorithm performance; when computing power is severely insufficient, time step expansion can be omitted to obtain basic generation results. It is understood that computing power can be understood as the processing capability of a computer chip, typically referring to I / O (input / output) throughput, processing and computation capabilities, etc. This invention does not limit the specific form of the chip. Alternatively, in one implementation, different computing power situations can be distinguished by setting computing power thresholds or computing power ranges. Alternatively, in another implementation, different computing power situations can also be distinguished based on the operating environment of the point cloud generation network, that is, based on the type of electronic device to which the point cloud generation method is applied. For example, to train a point cloud generation network capable of three feedback steps, there are three time steps: t0, t1, and t2. If the point cloud generation network runs on a mobile device or an ARM (a RISC microprocessor), where computing power is severely limited, the time step to be utilized is the earliest time step, i.e., time step t0. In this case, only one feedback step is needed to output the result. If the point cloud generation network runs on a PC without a GPU (graphics processing unit), where computing power is moderate, the time step to be utilized is any time step other than the earliest and latest time steps, i.e., time step t1. In this case, two feedback steps are needed to output the result. If the point cloud generation network runs on a server, where computing power is sufficient, the time step to be utilized is the latest time step, i.e., time step t2. In this case, three feedback steps are needed to output the result.

[0173] S203, update the value of the preset number threshold to the order value of the target time step in each time step minus 1;

[0174] There is no strict execution order between S201 and S202-S203. For example, S201 and S202-S203 can be executed in parallel, and S201 can be executed earlier or later than S202-S203.

[0175] The preset number threshold needs to be updated after the target time step is determined. The preset number threshold is updated from the initial default value to: the order value of the target time step in each time step minus 1. The initial default value has been introduced and will not be repeated here.

[0176] It should be noted that if the number of times the output content has been fed back reaches the preset threshold, then only the next sub-network will be input, and no feedback will be sent to the next time step feedback sub-network; if the number of times the output content has been fed back is less than the preset threshold, then both the next time step feedback sub-network and the next sub-network will be input.

[0177] Understandably, when the user updates the preset threshold value to k, after k feedbacks, no further time steps are taken for feedback; instead, the input is sent to the next sub-network. The entire point cloud generation network performs k+1 time steps. For example, if the current computing power is insufficient and the user sets the preset threshold to 1, meaning 1 feedback is required, then the entire point cloud generation network needs to perform 2 time steps.

[0178] S204, the point cloud data of the initial point cloud is input into a pre-trained point cloud generation network to obtain the output result of the point cloud generation network; wherein, the point cloud generation network includes at least one feedback sub-network with feedback function, each feedback sub-network is used to learn point cloud features using the acquired input content to obtain output content, and when a predetermined output content feedback condition is met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network continues to learn point cloud features based on the feedback content, and if there is a next sub-network, when a predetermined output content transmission condition is met, the current output content is input into the next sub-network; wherein, the designated sub-network corresponding to any feedback sub-network includes: the feedback sub-network and / or the sub-network located before the feedback sub-network;

[0179] It should be noted that the designated sub-network can also be the sub-network of the next time step. S204 is the same as S102 in the above embodiment, and will not be described again here.

[0180] S205, using the output result corresponding to the latest time step in each time step, determine the target point cloud for the target object;

[0181] The output of the latest time step among all time steps is determined as the target point cloud of the target object. For example, assuming there are two time steps t0 and t1, where t1 is later than t0, then the output of time step t1 is determined as the target point cloud of the target object.

[0182] As can be seen, this solution can feed back high-level feature data learned by the high-level network to the low-level network, thereby improving the semantic expressive power of the low-level feature data. In other words, it can transfer high-level semantic information learned by the high-level network to the low-level network to participate in the learning of low-level feature data, thus enhancing the semantic expressive power of the low-level feature data and ultimately improving the quality of the generated point cloud, i.e., improving the accuracy and completeness of the generated point cloud. Furthermore, compared to related technologies that improve the accuracy and completeness of point clouds by stacking a large number of network modules, this solution improves the accuracy and completeness of point clouds through a feedback mechanism, enabling a lightweight point cloud generation network and ultimately improving the overall network performance. Moreover, the time step can be selected based on the device's computing power, thereby ensuring the generation efficiency of the point cloud generation network.

[0183] Based on the above method embodiments, such as Figure 6 As shown, this embodiment of the invention also provides a point cloud generation device, which may include:

[0184] The acquisition module 610 is used to acquire point cloud data of the initial point cloud for the target object;

[0185] The input module 620 is used to input the point cloud data of the initial point cloud into a pre-trained point cloud generation network to obtain the output result of the point cloud generation network; wherein, the point cloud generation network includes at least one feedback sub-network with feedback function, each feedback sub-network is used to learn point cloud features using the acquired input content to obtain output content, and when a predetermined output content feedback condition is met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network continues to learn point cloud features based on the feedback content, and if there is a next sub-network, when a predetermined output content transmission condition is met, the current output content is input into the next sub-network; wherein, the designated sub-network corresponding to any feedback sub-network includes: the feedback sub-network and / or the sub-network located before the feedback sub-network;

[0186] The determination module 630 is used to determine the target point cloud for the target object using the output results.

[0187] Optionally, when the predetermined output content feedback conditions are met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the designated sub-network can continue to learn point cloud features based on the feedback content, including:

[0188] When the number of times the output content has been fed back is less than a preset threshold, the current output content is used as the feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network can continue to learn point cloud features based on the feedback content.

[0189] The step of inputting the current output content into the next sub-network when the predetermined output content transmission conditions are met includes:

[0190] If the current output content has not been passed to the next subnetwork in the feedback subnetwork, the current output content is input into the next subnetwork.

[0191] Optionally, when the predetermined output content feedback conditions are met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the designated sub-network can continue to learn point cloud features based on the feedback content, including:

[0192] When the number of times the output content has been fed back is less than a preset threshold, the current output content is used as the feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network can continue to learn point cloud features based on the feedback content.

[0193] The step of inputting the current output content into the next sub-network when the predetermined output content transmission conditions are met includes:

[0194] When the number of feedback attempts for the output content is less than a preset threshold, the current output content is input into the next sub-network.

[0195] Optionally, the output results generated by the point cloud generation network during training include the output results corresponding to each time step; the time step is a processing stage divided according to time sequence.

[0196] The output result corresponding to each time step is the output result generated by each sub-network in the point cloud generation network through deep point cloud feature learning using the data content corresponding to that time step; wherein, for any specified sub-network in each sub-network, the data content corresponding to that time step includes at least the latest output content of the content transmission end corresponding to that specified sub-network, and the feedback content of the feedback sub-network corresponding to that specified sub-network from the previous time step; the data content corresponding to other sub-networks in each sub-network includes at least the latest output content of the corresponding content transmission end; wherein, the content transmission end corresponding to any sub-network in the point cloud generation network is located before that sub-network and is used to transmit content to that sub-network;

[0197] The determining module includes:

[0198] The determination submodule is used to determine the target point cloud for the target object by using the output result corresponding to the latest time step in each time step.

[0199] Optionally, the device further includes:

[0200] The acquisition module is used to input the point cloud data of the initial point cloud into a pre-trained point cloud generation network and obtain the output result of the point cloud generation network before obtaining the time step to be utilized by the point cloud generation network during the operation of the electronic device, as the target time step; wherein, the time step to be utilized is any one of the various time steps, and the time step to be utilized is set based on the device type of the electronic device and / or the available computing power of the electronic device;

[0201] The update module is used to update the value of the preset number threshold to the order value of the target time step in each time step minus 1.

[0202] Optionally, the method of setting the time step to be utilized based on the device type of the electronic device includes:

[0203] If the electronic device is a device of the first type of device, the time step to be utilized is determined to be the earliest time step among all time steps;

[0204] If the electronic device is a device of the second type, the time step to be utilized is determined to be any time step other than the earliest time step and the latest time step among all time steps;

[0205] If the electronic device is a third type of device, the time step to be utilized is determined to be the latest time step among all time steps;

[0206] Among them, the computing power of the first type of equipment, the second type of equipment, and the third type of equipment gradually increases.

[0207] Optionally, after the point cloud generation network is trained, the parameter values ​​of the network parameters of the point cloud generation network are the same at different time steps.

[0208] Optionally, the designated subnetwork corresponding to any feedback subnetwork is specifically used to perform fusion processing on the currently received feedback content and the latest output content of the corresponding content transmission end each time feedback content is received, and to use the fused content obtained by the fusion processing to perform point cloud feature learning; wherein, the content transmission end corresponding to any subnetwork in the point cloud generation network is located before the subnetwork and is used to transmit content to the subnetwork.

[0209] Optionally, the fusion processing of the currently received feedback content and the latest output content from the corresponding content transmission end includes:

[0210] For each point in the latest output content of the corresponding content transmission terminal, among the points of the currently received feedback content, at least one neighboring point is determined. Using the feature data of at least one neighboring point and the feature data of the point, a specified feature fusion process is performed to obtain the fused data of the point.

[0211] Optionally, the fusion processing of the currently received feedback content and the latest output content from the corresponding content transmission end includes:

[0212] For each point in the latest output content of the corresponding content transmission end, calculate the attention weight of each point in the currently received feedback content for that point, and obtain the relevance weight matrix corresponding to that point;

[0213] For each point in the latest output content of the corresponding content transmission terminal, the feature data of each point in the currently received feedback content are weighted according to the relevance weight matrix corresponding to that point to obtain the feature data to be utilized for that point. The feature data of that point is then fused with the feature data to be utilized for that point to obtain the fused data for that point. Optionally, for each point in the latest output content of the corresponding content transmission terminal, at least one neighboring point is determined among the points in the currently received feedback content, including:

[0214] For each point in the latest output content of the corresponding content transmission end, the K nearest neighbor algorithm (KNN) is used to determine the K nearest points to the current point from each point in the currently received feedback content, and at least one neighboring point corresponding to the current point is obtained.

[0215] or,

[0216] For each point in the latest output content of the corresponding content transmission end, the attention weight of that point with each point in the currently received feedback content is determined using the relevance weight matrix corresponding to that point; based on the determined attention weight, at least one neighboring point is determined from each point in the currently received feedback content; wherein, the relevance weight matrix corresponding to that point is used to characterize the attention weight of that point with each point in the currently received feedback content.

[0217] Optionally, the point cloud generation network is obtained by training the network based on distance loss or adversarial loss; wherein the distance loss includes chamfered distance (CD) loss or plain distance (EMD) loss.

[0218] This invention provides an autonomous driving device, including: a data acquisition module and a processing module;

[0219] The acquisition module is used to generate an initial point cloud for the target object;

[0220] The processing module is used to generate a target point cloud for the target object using point cloud data of the initial point cloud for the target object, according to any of the point cloud generation methods described above.

[0221] Optionally, the processing module is further configured to, upon receiving an obstacle avoidance task or navigation task for the target object, execute the obstacle avoidance task or navigation task for the target object based on the target point cloud of the target object and using the traveling component.

[0222] Optionally, the acquisition module includes a depth camera;

[0223] The depth camera is used to acquire images of the target object and to construct an initial point cloud for the target object using the acquired images.

[0224] The autonomous driving device also includes a driving component that drives the autonomous driving device to move based on the target point cloud of the target object.

[0225] Optionally, the acquisition module includes a lidar;

[0226] The lidar is used to collect radar data from the target object and to construct an initial point cloud for the target object using the collected radar data.

[0227] The autonomous driving device also includes a driving component that drives the autonomous driving device to move based on the target point cloud of the target object.

[0228] This invention also provides an autonomous driving device, such as... Figure 7 As shown, it includes: a data acquisition module 710 and a processing module 730;

[0229] The acquisition module 710 is used to generate an initial point cloud for the target object;

[0230] The processing module 730 is used in any of the above point cloud generation methods to generate a target point cloud for the target object using point cloud data of the initial point cloud for the target object.

[0231] Optionally, the processing module 730 is further configured to, upon receiving an obstacle avoidance task or navigation task for the target object, execute the obstacle avoidance task or navigation task for the target object based on the target point cloud of the target object.

[0232] Optionally, the acquisition module 710 includes a depth camera;

[0233] The depth camera is used to acquire images of the target object and to construct an initial point cloud for the target object using the acquired images.

[0234] The autonomous driving device also includes a traveling component 720, which drives the autonomous driving device to travel based on the target point cloud of the target object.

[0235] The processing module 730 is further configured to drive the autonomous driving device to move by controlling the moving component 720 based on the target point cloud of the target object.

[0236] During the process of the acquisition module 710 generating the initial point cloud for the target object, the traveling component 720 can drive the autonomous driving device to travel around the target object to acquire complete point cloud information, or it can drive the autonomous driving device to approach the target object to acquire more detailed point cloud data.

[0237] Optionally, the acquisition module 710 includes a lidar;

[0238] The lidar is used to collect radar data from the target object and to construct an initial point cloud for the target object using the collected radar data.

[0239] The autonomous driving device also includes a traveling component 720, which drives the autonomous driving device to travel based on the target point cloud of the target object.

[0240] The following will use a robotic vacuum cleaner as an example to introduce the functions of the autonomous driving device:

[0241] Robotic vacuum cleaners are equipped with TOF cameras (ToF Camera, Full Resolution Depth Camera). Due to factors such as object occlusion, sensor resolution (i.e., the resolution of the TOF camera), object material, and environmental conditions, the initial point cloud generated from the image data collected by the TOF camera inevitably suffers from issues such as missing points, low resolution, and noise. Therefore, to improve the accuracy and completeness of the point cloud, the robotic vacuum cleaner can use the TOF camera to collect an initial point cloud of the surrounding environment. Then, the processing module 730 processes this data to repair invisible areas of the target object, resulting in a more complete and accurate target point cloud. This target point cloud allows the robotic vacuum cleaner to more accurately predict obstacles or impassable areas, thus achieving more efficient obstacle avoidance.

[0242] It is understood that TOF cameras can collect image data of obstacles of any material, so this invention does not limit the material of the obstacle; and since the processing module can use the above-mentioned point cloud generation method to make more complete and accurate corrections for the invisible areas of the target object, the robot vacuum cleaner can generate a complete and highly accurate target point cloud of the target object regardless of the point cloud missing, low resolution, noise, or other problems caused by factors such as self-occlusion of the target object, sensor resolution, object material, and environment. This allows for efficient obstacle avoidance during task execution based on the target point cloud.

[0243] The autonomous driving device provided by the embodiments of the present invention can improve the quality of generated point clouds, that is, improve the accuracy and completeness of generated point clouds. Furthermore, compared with related technologies that improve the accuracy and completeness of point clouds by stacking a large number of network modules, the point cloud generation network used in the autonomous driving device can achieve lightweight operation while improving the accuracy and completeness of point clouds through a feedback mechanism, thereby ultimately improving the overall network performance.

[0244] This invention also provides an electronic device, such as... Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, communication interface 802, and memory 803 communicate with each other via the communication bus 804.

[0245] Memory 803 is used to store computer programs;

[0246] The processor 801 is used to execute the program stored in the memory 803 to implement the point cloud generation method steps.

[0247] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0248] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0249] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0250] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0251] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the point cloud generation methods described above.

[0252] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the point cloud generation methods described above.

[0253] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0254] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0255] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0256] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A point cloud generation method, characterized in that, Applied to electronic devices, the method includes: Obtain the point cloud data of the initial point cloud for the target object; The point cloud data of the initial point cloud is input into a pre-trained point cloud generation network to obtain the output result of the point cloud generation network. The point cloud generation network includes at least one feedback sub-network with feedback functionality. Each feedback sub-network is used to learn point cloud features using the acquired input content to obtain output content. When a predetermined output content feedback condition is met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the designated sub-network continues to learn point cloud features based on the feedback content. If a next sub-network exists, when a predetermined output content transmission condition is met, the current output content is input into the next sub-network. The designated sub-network corresponding to any feedback sub-network includes: the feedback sub-network and / or the sub-network preceding the feedback sub-network. Using the output results, the target point cloud for the target object is determined; Specifically, the designated subnetwork corresponding to any feedback subnetwork is used to fuse the currently received feedback content and the latest output content of the corresponding content transmission end each time feedback content is received, and to use the fused content obtained by the fusion process to learn point cloud features; wherein, the content transmission end corresponding to any subnetwork in the point cloud generation network is located before the subnetwork and is used to transmit content to the subnetwork. The process of fusing the currently received feedback content with the latest output content from the corresponding content transmission end includes: For each point in the latest output content of the corresponding content transmission terminal, among the points of the currently received feedback content, at least one neighboring point is determined. Using the feature data of at least one neighboring point and the feature data of the point, a specified feature fusion process is performed to obtain the fused data of the point.

2. The method according to claim 1, characterized in that, When the predetermined output content feedback conditions are met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the designated sub-network can continue to learn point cloud features based on the feedback content, including: When the number of times the output content has been fed back is less than a preset threshold, the current output content is used as the feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network can continue to learn point cloud features based on the feedback content. The step of inputting the current output content into the next sub-network when the predetermined output content transmission conditions are met includes: If the current output content has not been passed to the next subnetwork in the feedback subnetwork, the current output content is input into the next subnetwork.

3. The method according to claim 1, characterized in that, When the predetermined output content feedback conditions are met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the designated sub-network can continue to learn point cloud features based on the feedback content, including: When the number of times the output content has been fed back is less than a preset threshold, the current output content is used as the feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network can continue to learn point cloud features based on the feedback content. The step of inputting the current output content into the next sub-network when the predetermined output content transmission conditions are met includes: When the number of feedback attempts for the output content is less than a preset threshold, the current output content is input into the next sub-network.

4. The method according to claim 2, characterized in that, The output generated by the point cloud generation network during training includes the output at each time step; the time step is a processing stage divided according to time sequence. The output result corresponding to each time step is the output result generated by each sub-network in the point cloud generation network through deep point cloud feature learning using the data content corresponding to that time step; wherein, for any specified sub-network in each sub-network, the data content corresponding to that time step includes at least the latest output content of the content transmission end corresponding to that specified sub-network, and the feedback content of the feedback sub-network corresponding to that specified sub-network from the previous time step; the data content corresponding to other sub-networks in each sub-network includes at least the latest output content of the corresponding content transmission end; wherein, the content transmission end corresponding to any sub-network in the point cloud generation network is located before that sub-network and is used to transmit content to that sub-network; Determining the target point cloud for the target object using the output results includes: The target point cloud for the target object is determined by using the output result corresponding to the latest time step in each time step.

5. The method according to claim 4, characterized in that, Before inputting the point cloud data of the initial point cloud into a pre-trained point cloud generation network to obtain the output of the point cloud generation network, the method further includes: The point cloud generation network is obtained as the target time step during the operation of the electronic device. The target time step is any one of the various time steps and is set based on the device type of the electronic device and / or the available computing power of the electronic device. The value of the preset number threshold is updated to the order value of the target time step in each time step minus 1.

6. The method according to claim 5, characterized in that, The method of setting the time step to be used based on the device type of the electronic device includes: If the electronic device is a device of the first type of device, the time step to be utilized is determined to be the earliest time step among all time steps; If the electronic device is a device of the second type, the time step to be utilized is determined to be any time step other than the earliest time step and the latest time step among all time steps; If the electronic device is a third type of device, the time step to be utilized is determined to be the latest time step among all time steps; Among them, the computing power of the first type of equipment, the second type of equipment, and the third type of equipment gradually increases.

7. The method according to claim 4, characterized in that, After training, the point cloud generation network has the same parameter values ​​at different time steps.

8. The method according to claim 1, characterized in that, The process of fusing the currently received feedback content with the latest output content from the corresponding content transmission end includes: For each point in the latest output content of the corresponding content transmission end, calculate the attention weight of each point in the currently received feedback content for that point, and obtain the relevance weight matrix corresponding to that point; For each point in the latest output content of the corresponding content transmission terminal, the feature data of each point in the currently received feedback content are weighted according to the relevance weight matrix corresponding to that point to obtain the feature data to be used for that point. The feature data of that point is then fused with the feature data to be used for that point to obtain the fused data of that point.

9. The method according to claim 1, characterized in that, For each point in the latest output content of the corresponding content transmission end, among all points in the currently received feedback content, determine at least one neighboring point corresponding to that point, including: For each point in the latest output content of the corresponding content transmission end, the K nearest neighbor algorithm (KNN) is used to determine the K nearest points to the current point from each point in the currently received feedback content, and at least one neighboring point corresponding to the current point is obtained. or, For each point in the latest output content of the corresponding content transmission terminal, the attention weight between that point and each point in the currently received feedback content is determined using the relevance weight matrix corresponding to that point; based on the determined attention weight, at least one neighboring point is determined from each point in the currently received feedback content; wherein, the relevance weight matrix corresponding to that point is used to characterize the attention weight between that point and each point in the currently received feedback content.

10. The method according to any one of claims 1-7, characterized in that, The point cloud generation network is obtained by training the network based on distance loss or adversarial loss; wherein, the distance loss includes chamfer distance CD loss or plain distance EMD loss.

11. A point cloud generation device, characterized in that, Applied to electronic devices, the device includes: The acquisition module is used to acquire point cloud data of the initial point cloud for the target object; An input module is used to input the point cloud data of the initial point cloud into a pre-trained point cloud generation network to obtain the output result of the point cloud generation network; wherein, the point cloud generation network includes at least one feedback sub-network with feedback function, each feedback sub-network is used to learn point cloud features using the acquired input content to obtain output content, and when a predetermined output content feedback condition is met, the current output content is used as feedback content and fed back to the corresponding designated sub-network so that the corresponding designated sub-network continues to learn point cloud features based on the feedback content, and if there is a next sub-network, when a predetermined output content transmission condition is met, the current output content is input into the next sub-network; wherein, the designated sub-network corresponding to any feedback sub-network includes: the feedback sub-network and / or the sub-network located before the feedback sub-network; A determination module is used to determine the target point cloud for the target object using the output results; Specifically, the designated subnetwork corresponding to any feedback subnetwork is used to fuse the currently received feedback content and the latest output content of the corresponding content transmission end each time feedback content is received, and to use the fused content obtained by the fusion process to learn point cloud features; wherein, the content transmission end corresponding to any subnetwork in the point cloud generation network is located before the subnetwork and is used to transmit content to the subnetwork. The process of fusing the currently received feedback content with the latest output content from the corresponding content transmission end includes: For each point in the latest output content of the corresponding content transmission terminal, among the points of the currently received feedback content, at least one neighboring point is determined. Using the feature data of at least one neighboring point and the feature data of the point, a specified feature fusion process is performed to obtain the fused data of the point.

12. An automatic driving device, characterized in that, include: Acquisition module and processing module; The acquisition module is used to generate an initial point cloud for the target object; A processing module is configured to generate a target point cloud for the target object using point cloud data of an initial point cloud for the target object, according to any one of claims 1-10.

13. The automatic driving device according to claim 12, characterized in that, The processing module is further configured to, upon receiving an obstacle avoidance task or navigation task for the target object, execute the obstacle avoidance task or navigation task for the target object based on the target point cloud of the target object.

14. The automatic driving device according to claim 12 or 13, characterized in that, The acquisition module includes a depth camera; The depth camera is used to acquire images of the target object and to construct an initial point cloud for the target object using the acquired images. The autonomous driving device also includes a driving component that drives the autonomous driving device to move based on the target point cloud of the target object.

15. The automatic driving device according to claim 12 or 13, characterized in that, The acquisition module includes a lidar; The lidar is used to collect radar data from the target object and to construct an initial point cloud for the target object using the collected radar data. The autonomous driving device also includes a driving component that drives the autonomous driving device to move based on the target point cloud of the target object.

Images