Point cloud completion method based on semantic information, model training method and device

Abstract

The present disclosure proposes a point cloud completion method based on semantic information, a model training method and a device, wherein the point cloud completion method includes: performing image processing on a binocular image to be processed to obtain semantic information of a target object in the binocular image to be processed and a first depth image of the binocular image to be processed; performing a coarse-grained completion operation on the residual point cloud to be processed based on the semantic information, and projecting the point cloud after the coarse-grained completion operation into a two-dimensional space to obtain a second depth image corresponding to the residual point cloud; performing a fusion operation on the first depth image and the second depth image to obtain a depth completion image; and back-projecting the depth completion image to obtain a completed three-dimensional point cloud containing the target object. By implementing the embodiments of the present disclosure, the point cloud completion effect can be significantly improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field, and in particular to a point cloud completion method based on semantic information, a model training method and a device. Background Art

[0002] Currently, welding robots are increasingly used in the industrial field, and weld recognition technology, as the "eyes" of welding robots, is one of the key technologies to achieve efficient welding. In related technologies, weld recognition technology mainly relies on 3D (three-dimensional) camera sensors, such as 3D cameras based on line laser scanning, which are widely used due to the good singleness and anti-interference of lasers. Common line laser 3D cameras are usually equipped with a laser scanner and a pair of binocular cameras, which estimate the depth of the workpiece through traditional binocular matching combined with line laser features.

[0003] However, in certain scenarios, such as when the surface of the welded workpiece is dark, the laser may be absorbed, resulting in point cloud holes in the 3D camera imaging results, making it impossible for the robot to correctly identify the weld, resulting in welding failure. Therefore, how to solve the point cloud hole problem has become an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a point cloud completion method based on semantic information, a model training method and a device.

[0005] According to a first aspect of an embodiment of the present disclosure, a point cloud completion method based on semantic information is provided, comprising:

[0006] Performing image processing on the binocular image to be processed to obtain semantic information of the target object in the binocular image to be processed and a first depth image of the binocular image to be processed;

[0007] Based on the semantic information, a coarse-grained completion operation is performed on the residual point cloud to be processed, and the point cloud after the coarse-grained completion operation is projected into a two-dimensional space to obtain a second depth image corresponding to the residual point cloud; the residual point cloud to be processed is a point cloud including the target object, and the target object in the binocular image to be processed and the target object in the residual point cloud to be processed are the same target object;

[0008] Performing a fusion operation on the first depth image and the second depth image to obtain a depth completion image;

[0009] Back-projecting the depth completion image to obtain a completed three-dimensional point cloud containing the target object.

[0010] According to a second aspect of an embodiment of the present disclosure, a point cloud completion model training method is provided, wherein the point cloud completion model includes a binocular depth estimation network and a depth fusion network, and the training method includes:

[0011] Inputting the binocular image sample into the binocular depth estimation network for semantic analysis and depth information extraction to obtain semantic information of the target object sample in the binocular image sample and a first depth image of the binocular image sample;

[0012] Based on the semantic information, a coarse-grained completion operation is performed on the residual defect cloud sample, and the point cloud after the coarse-grained completion operation is projected into a two-dimensional space to obtain a second depth image corresponding to the residual defect cloud sample; the residual defect cloud sample is a point cloud including the target object sample, and the target object sample in the binocular image sample and the target object sample in the residual defect cloud sample are the same target object;

[0013] Determine a first model loss value based on the semantic information, the first depth image, and the second depth image in combination with a model loss function adopted by the binocular depth estimation network;

[0014] Inputting the first depth image and the second depth image into the depth fusion network for fusion operation to obtain a first depth completion image;

[0015] Back-projecting the first depth completion image to obtain a completed three-dimensional point cloud containing the target object sample;

[0016] Determine a second model loss value based on a completed three-dimensional point cloud containing the target object sample, a complete point cloud associated with the residual point cloud sample, and a target object sample label corresponding to the point cloud;

[0017] The binocular depth estimation network is model trained based on the first model loss value, and the depth fusion network is model trained based on the second model loss value.

[0018] According to a third aspect of an embodiment of the present disclosure, an end-to-end point cloud completion method is provided, including:

[0019] Acquire a binocular image to be processed and a residual spot cloud to be processed, wherein the residual spot cloud to be processed is a point cloud including a target object, and the target object in the binocular image to be processed and the target object in the residual spot cloud to be processed are the same target object;

[0020] The binocular image to be processed and the residual point cloud to be processed are input into a point cloud completion model to obtain a completed three-dimensional point cloud containing the target object; wherein the point cloud completion model is a model trained based on the method described in the second aspect above.

[0021] According to a fourth aspect of an embodiment of the present disclosure, a point cloud completion device based on semantic information is provided, comprising:

[0022] A semantic analysis module, used for performing image processing on the binocular image to be processed to obtain semantic information of the target object in the binocular image to be processed and a first depth image of the binocular image to be processed;

[0023] A completion preprocessing module, configured to perform a coarse-grained completion operation on the residual point cloud to be processed based on the semantic information, and project the point cloud after the coarse-grained completion operation into a two-dimensional space to obtain a second depth image corresponding to the residual point cloud; the residual point cloud to be processed is a point cloud including the target object, and the target object in the binocular image to be processed and the target object in the residual point cloud to be processed are the same target object;

[0024] A fusion module, configured to perform a fusion operation on the first depth image and the second depth image to obtain a depth completion image;

[0025] The back-projection module is used to back-project the depth completion image to obtain a completed three-dimensional point cloud containing the target object.

[0026] According to a fifth aspect of an embodiment of the present disclosure, a point cloud completion model training device is provided, wherein the point cloud completion model includes a binocular depth estimation network and a depth fusion network, and the training device includes:

[0027] A first acquisition module is used to input the binocular image sample into the binocular depth estimation network for semantic analysis and depth information extraction, so as to obtain the semantic information of the target object sample in the binocular image sample and the first depth image of the binocular image sample;

[0028] a completion preprocessing module, configured to perform a coarse-grained completion operation on the residual defect cloud sample based on the semantic information, and project the point cloud after the coarse-grained completion operation into a two-dimensional space to obtain a second depth image corresponding to the residual defect cloud sample; the residual defect cloud sample is a point cloud including the target object sample, and the target object sample in the binocular image sample and the target object sample in the residual defect cloud sample are the same target object;

[0029] A first determination module, configured to determine a first model loss value based on the semantic information, the first depth image, and the second depth image in combination with a model loss function adopted by the binocular depth estimation network;

[0030] A fusion module, used for inputting the first depth image and the second depth image into the depth fusion network for fusion operation to obtain a first depth completion image;

[0031] A back-projection module, configured to back-project the first depth completion image to obtain a completed three-dimensional point cloud containing a sample of the target object;

[0032] A second determination module is used to determine a second model loss value based on the completed three-dimensional point cloud containing the target object sample, the complete point cloud associated with the residual point cloud sample, and the target object sample label corresponding to the point cloud;

[0033] A training module is used to perform model training on the binocular depth estimation network based on the first model loss value, and to perform model training on the depth fusion network based on the second model loss value.

[0034] According to a sixth aspect of an embodiment of the present disclosure, an end-to-end point cloud completion device is provided, including:

[0035] A first acquisition module is used to acquire a binocular image to be processed and a residual spot cloud to be processed, wherein the residual spot cloud to be processed is a point cloud including a target object, and the target object in the binocular image to be processed and the target object in the residual spot cloud to be processed are the same target object;

[0036] The second acquisition module is used to input the binocular image to be processed and the residual point cloud to be processed into a point cloud completion model to obtain a completed three-dimensional point cloud containing the target object; wherein the point cloud completion model is a model trained based on the method described in the second aspect above.

[0037] According to the seventh aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods described in the first, second and third aspects above.

[0038] According to an eighth aspect of an embodiment of the present disclosure, a storage medium is provided, wherein the storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the methods described in the first, second and third aspects above.

[0039] According to a ninth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method described in the first, second and third aspects are implemented.

[0040] According to the technical solution disclosed in the present invention, by combining semantic information, the depth image of the binocular image to be processed and the depth image corresponding to the residual point cloud, a complete and accurate point cloud completion can be achieved, which can significantly improve the accuracy of point cloud completion and the effect of point cloud completion. For example, it can be applied to weld seam recognition scenarios, which can improve the weld seam recognition effect, thereby improving the generalization ability, welding efficiency and quality of welding robots.

[0041] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0043] Figure 1A A flowchart of a point cloud completion method based on semantic information provided by an embodiment of the present disclosure;

[0044] Figure 1B An example diagram of a point cloud void effect occurring in a weld of a workpiece provided by an embodiment of the present disclosure;

[0045] Figure 1C The effect diagram after the incomplete defect cloud provided by the embodiment of the present disclosure is completed;

[0046] Figure 2 A flowchart of a point cloud completion method based on semantic information provided by an embodiment of the present disclosure;

[0047] Figure 3 A flowchart of a method for training a binocular depth estimation network provided by an embodiment of the present disclosure;

[0048] Figure 4 A schematic diagram of the point cloud completion preprocessing process provided by an embodiment of the present disclosure;

[0049] Figure 5 A flowchart of a training method for a deep fusion network provided in an embodiment of the present disclosure;

[0050] Figure 6 A flowchart of a point cloud completion model training method provided in an embodiment of the present disclosure;

[0051] Figure 7 This is a flowchart of a point cloud completion model training method provided in an embodiment of the present disclosure;

[0052] Figure 8 A schematic diagram of a flow chart of an end-to-end point cloud completion method provided by an embodiment of the present disclosure;

[0053] Fig. 9 A block diagram of a point cloud completion device based on semantic information provided by an embodiment of the present disclosure;

[0054] Fig.10 A block diagram of a point cloud completion model training device provided in an embodiment of the present disclosure;

[0055] Fig.11 A block diagram of an end-to-end point cloud completion device provided by an embodiment of the present disclosure;

[0056] Fig.12 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0058] The embodiments of the present disclosure are not exhaustive, but are merely illustrative of some embodiments, and are not intended to be a specific limitation on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementation methods in a certain embodiment can be arbitrarily combined; in addition, the embodiments can be arbitrarily combined, for example, some or all of the steps of different embodiments can be arbitrarily combined, and a certain embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0059] The prefixes such as "first" and "second" in the embodiments of the present disclosure are only used to distinguish different description objects, and do not constitute restrictions on the position, order, priority, quantity or content of the description objects. The statement of the description object refers to the description in the context of the claims or embodiments, and should not constitute unnecessary restrictions due to the use of prefixes. For example, if the description object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields", and the "first" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the description object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of description objects is not limited by the ordinal number, and can be one or more. Taking the "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes may be the same or different. For example, if the description object is "device", then the "first device" and the "second device" may be the same device or different devices, and their types may be the same or different. For another example, if the description object is "information", then the "first information" and the "second information" may be the same information or different information, and their contents may be the same or different.

[0060] In each embodiment of the present disclosure, if there is no special explanation and logical conflict, the terms and / or descriptions between the embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form a new embodiment according to their inherent logical relationship. The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure.

[0061] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0062] It should also be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0063] It is worth noting that in the embodiments of the present disclosure, certain software, components, models, etc. that already exist in the industry may be mentioned. They should be considered as exemplary and their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0064] Currently, welding robots are increasingly used in the industrial field, and weld recognition technology, as the "eyes" of welding robots, is one of the key technologies to achieve efficient welding. In related technologies, weld recognition technology mainly relies on 3D (three-dimensional) camera sensors, such as 3D cameras based on line laser scanning, which are widely used due to the good singleness and anti-interference of lasers. Common line laser 3D cameras are usually equipped with a laser scanner and a pair of binocular cameras, which estimate the depth of the workpiece through traditional binocular matching combined with line laser features.

[0065] However, in certain scenarios, such as when the surface of the welded workpiece is dark, the laser may be absorbed, resulting in point cloud holes in the 3D camera imaging results, making it impossible for the robot to correctly identify the weld, resulting in welding failure. Therefore, how to solve the point cloud hole problem has become an urgent problem to be solved.

[0066] Based on the above problems, the present disclosure provides a point cloud completion method based on semantic information, a point cloud completion model training method and device. Complete and accurate workpiece depth information and semantic information can be predicted to complete the point cloud, which can significantly improve the point cloud completion effect, thereby significantly improving the generalization ability, welding efficiency and quality of the welding robot.

[0067] The following describes a point cloud completion method based on semantic information, a point cloud completion model training method and a device according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0068] Figure 1A Schematic diagram of the flow of the point cloud completion method based on semantic information provided by the embodiment of the present disclosure. Figure 1A As shown, the point cloud completion method based on semantic information may include but is not limited to the following steps.

[0069] In step 101, image processing is performed on a binocular image to be processed to obtain semantic information of a target object in the binocular image to be processed and a first depth image of the binocular image to be processed.

[0070] In some embodiments, semantic segmentation technology can be used to perform semantic analysis on the binocular image to be processed to obtain semantic information of the target object in the binocular image to be processed. Exemplarily, a neural network model with semantic segmentation function, such as a binocular depth estimation network, can be used to perform semantic analysis on the binocular image to be processed to obtain semantic information of the target object in the binocular image to be processed.

[0071] In some embodiments, the binocular image to be processed may be a binocular image including a target object. In some embodiments, the binocular image to be processed may be an image obtained by photographing the target object with a binocular camera. In some embodiments, the target object may be a weld in a welding workpiece, but is not limited thereto. Exemplarily, the first image and the second image in the binocular image may both be RGB images, but are not limited thereto.

[0072] In some embodiments, the binocular image to be processed includes a first image and a second image. For example, the binocular image to be processed may be a left and right viewpoint image of the same target object scene taken by a binocular camera, and a disparity map is obtained by using a stereo matching algorithm. Based on the disparity map, a first depth image of the binocular image to be processed is obtained.

[0073] It should be noted that, in some embodiments, other depth image generation methods can also be used to generate the first depth image of the binocular image to be processed. For example, a neural network can be used to achieve this. For example, the binocular image to be processed is input into a binocular depth estimation network to obtain the first depth image of the binocular image to be processed, but it is not limited to this. The present disclosure does not make specific limitations on this and will not go into details.

[0074] In step 102 , a coarse-grained completion operation is performed on the residual point cloud to be processed based on semantic information, and the point cloud after the coarse-grained completion operation is projected into a two-dimensional (2D) space to obtain a second depth image corresponding to the residual point cloud to be processed.

[0075] In some embodiments, the semantic information may include the location of the target object in the binocular image to be processed. For example, taking the target object as a weld in a welding workpiece as an example, the semantic information may include the location of the weld in the binocular image to be processed.

[0076] In some embodiments, after obtaining the semantic information of the target object, a coarse-grained completion operation can be performed on the residual point cloud to be processed based on the semantic information. Exemplarily, point cloud data belonging to the same direction as the target object can be found based on the semantic information, because the point cloud with the same semantic information has a high degree of structural similarity locally, so the local structural similarity can be used to perform a coarse-grained completion operation on the incomplete part (or also called a hole area, or a point cloud hole area) in the residual point cloud to be processed. Exemplarily, taking the weld in the welded workpiece as an example, the point cloud data belonging to the same weld can be found based on the semantic information of the weld, and the coarse-grained completion operation is performed on the incomplete part of the weld point cloud based on the found point cloud data. The point cloud after the coarse-grained completion operation is projected into the 2D space to obtain a second depth image. In this embodiment, by using a point cloud with high local structural similarity based on semantic information to roughly complete the incomplete part, point cloud information of similar structure can be obtained without the complex operation of reconstructing the surface, and semantic information can be provided to improve the accuracy of point cloud completion.

[0077] In some embodiments, the residual defect cloud to be processed may be a point cloud including a target object, and the target object in the binocular image to be processed and the target object in the residual defect cloud to be processed may be the same target object. Exemplarily, the target object may be a weld in a welding workpiece, and a laser scanner may be used to perform laser scanning on the weld in the welding workpiece to obtain a residual defect cloud to be processed; and a binocular camera may be used to photograph the weld in the welding workpiece to obtain a binocular image to be processed.

[0078] In step 103, a fusion operation is performed on the first depth image and the second depth image to obtain a depth completion image.

[0079] In some embodiments, the one-to-one correspondence between pixels in the first depth image and the second depth image can be found based on the calibration information, and a fusion operation can be performed based on the correspondence to obtain a depth-completed image. The calibration information may include but is not limited to parameter information of a device that captures the binocular image to be processed (such as a binocular camera) and parameter information of a device that captures the residual cloud to be processed (such as a laser scanner).

[0080] In some embodiments, the above-mentioned "fusion operation" may be an image superposition process, such as directly adding two depth images, but not limited thereto. For example, the two depth images may be fused using their respective weights, and the fusion process may be, for example, a weighted summation operation. In some embodiments, the weight values ​​of the two depth images may be trained by deep learning, and the two depth images may be fused based on the trained weight values, and the fused image obtained after fusion may be used as the depth completion image.

[0081] In some embodiments, the first depth image may be a depth image obtained by processing the binocular image to be processed based on a pre-trained binocular depth estimation network, and the binocular depth estimation network may make the depth information of the first depth image and the second depth image globally consistent, and may make the first depth image and the second depth image have the same geometric structure consistency of the semantic information point cloud. Based on this characteristic, after obtaining the first depth image and the second depth image, the first depth image and the second depth image are fused, that is, the first depth image is used to adjust the area information of the second depth image that has undergone a coarse-grained completion operation, so as to obtain a complete and accurate depth completion image of the target object.

[0082] In step 104, the depth completion image is back-projected to obtain a completed three-dimensional point cloud containing the target object.

[0083] In some embodiments, the above “back projection” may refer to mapping from 2D (two-dimensional) space to 3D (three-dimensional) space. Exemplarily, after obtaining a depth completion image, the depth completion image may be mapped from 2D space to 3D space to obtain a completed three-dimensional point cloud containing the target object.

[0084] In some embodiments, the point cloud completion method based on semantic information involved in the present disclosure can be applied to weld identification tasks. For example, in the weld identification process, the point cloud completion method based on semantic information involved in the present disclosure can be used to complete the residual point cloud. Figure 1B As shown in FIG. 1 , area F is an example of a point cloud void effect in the 3D camera imaging result when the weld of the workpiece is photographed by a 3D camera. Figure 1C As shown, it is the effect diagram after the residual point cloud is completed by using the semantic information-based point cloud completion method involved in the present disclosure. It can be seen that by using the point cloud completion method involved in the present disclosure, complete and accurate weld point cloud information can be obtained, thereby significantly improving the generalization ability, welding efficiency and quality of the welding robot.

[0085] In the above embodiment, by combining semantic information, the depth image of the binocular image to be processed and the depth image corresponding to the residual point cloud, a complete and accurate point cloud completion can be achieved, which can significantly improve the accuracy of point cloud completion and improve the point cloud completion effect. For example, it can be applied to weld seam recognition scenarios to improve weld seam recognition effects, thereby improving the generalization ability, welding efficiency and quality of welding robots.

[0086] Figure 2 Schematic diagram of the flow of the point cloud completion method based on semantic information provided by the embodiment of the present disclosure. Figure 2 As shown, the point cloud completion method based on semantic information may include but is not limited to the following steps.

[0087] In step 201, semantic analysis and depth information extraction are performed on a binocular image to be processed based on a pre-trained binocular depth estimation network to obtain semantic information of a target object in the binocular image to be processed and a first depth image of the binocular image to be processed.

[0088] Exemplarily, the binocular image to be processed can be input into a pre-trained binocular depth estimation network for semantic analysis and depth information extraction to obtain semantic information of the target object in the binocular image to be processed and a first depth image of the binocular image to be processed. In some embodiments, the input of the binocular depth estimation network can be the binocular image to be processed, and the output of the binocular depth estimation network can include the semantic information of the target object and the first depth image, and the binocular depth estimation network has learned the mapping relationship between the binocular image and the semantic information of the target object and the first depth image.

[0089] In some embodiments, the above binocular depth estimation network can be pre-trained, such as Figure 3 As shown, the training method of the above binocular depth estimation network may include but is not limited to the following steps:

[0090] In step 301, a binocular image sample is input into a binocular depth estimation network for semantic analysis and depth information extraction to obtain semantic information of a target object sample in the binocular image sample and a first depth image of the binocular image sample.

[0091] In step 302, a first model loss value is determined based on the semantic information of the target object sample, the first depth image of the binocular image sample, and the second depth image corresponding to the residual cloud sample, in combination with the model loss function adopted by the binocular depth estimation network.

[0092] In some embodiments, the residual defect cloud sample may be a point cloud including a target object sample, and the target object sample in the binocular image sample and the target object sample in the residual defect cloud sample are the same target object.

[0093] Exemplarily, the semantic information of the target object sample, the first depth image of the binocular image sample, and the second depth image corresponding to the residual cloud sample can be substituted into the model loss function adopted by the binocular depth estimation network to calculate the loss value, and the calculated loss value can be determined as the first model loss value of the binocular depth estimation network.

[0094] In some embodiments, the model loss function used by the above-mentioned binocular depth estimation network can be based on a first loss function, a second loss function and a third loss function. Among them, the first loss function is used to constrain the scale of the first depth image and the scale of the second depth image to be consistent; the second loss function is combined with semantic information to constrain the geometric structure consistency of the same semantic pixels in the first depth image and the second depth image in the local space; the third loss function is used to constrain the first depth image and the second depth image to be consistent in global geometry.

[0095] Exemplarily, the model loss function used by the binocular depth estimation network can be expressed as:

[0096]

[0097] in, The model loss function used by the stereo depth estimation network; is the first loss function, is the weight of the first loss function; is the second loss function, is the weight of the second loss function; is the third loss function, is the weight of the third loss function. For example, the first loss function The expression can be:

[0098]

[0099] in, is the first depth image The depth value of the pixel, is the first The depth value of the pixel, is the number of pixels in the first depth image and the second depth image. The first loss function It can be used to constrain the consistency of the depth scales of the first depth image and the second depth image.

[0100] Exemplarily, the second loss function The expression can be:

[0101]

[0102] in, is the first The depth value of the pixel, is the first The depth value of pixels; Can indicate the The depth value of the pixel is related to the surroundingr The depth values ​​of pixels with the same semantics within the distance range should be as consistent as possible, and Belongs to the semantic point, N is the number of semantic points. Represents the weight, which is calculated as: , .in, , Respectively expressed as The position of the pixel in the second depth image, , Respectively expressed as The position of a pixel in the second depth image; is the standard deviation of the Gaussian function. The second loss function Combined with semantic information, it is used to constrain the geometric structure consistency of pixels with the same semantics in the local space.

[0103] Exemplarily, the third loss function The expression can be:

[0104]

[0105] in, is the first depth image Normal constraints for points with pixels, is the first The third loss function It is mainly used to constrain the global geometric structure consistency of the first depth image and the second depth image.

[0106] It can be seen that the advantage of the model loss function constraint adopted by the stereo depth estimation network is that while constraining the global consistency of the depth information, it ensures the geometric structure consistency of the point cloud with the same semantic information, and further improves the completion accuracy of the hole area of ​​the point cloud.

[0107] In some embodiments, the second depth image corresponding to the above-mentioned residual spot cloud sample can be obtained based on the semantic information of the target object sample. Exemplarily, a coarse-grained completion operation can be performed on the residual spot cloud sample based on the semantic information of the target object sample, and the point cloud after the coarse-grained completion operation is projected into 2D space to obtain the second depth image corresponding to the residual spot cloud sample. The optional implementation method is similar to the optional implementation method of "obtaining the second depth image corresponding to the residual spot cloud" in this article. Please refer to the relevant description of how to obtain the second depth image corresponding to the residual spot cloud in this article, which will not be repeated here.

[0108] In step 303, a model training is performed on the binocular depth estimation network based on the first model loss value.

[0109] Exemplarily, a gradient descent algorithm can be used to perform gradient descent processing on the first model loss value, and the parameters in the binocular depth estimation network can be updated according to the result of the gradient descent processing until the model training end condition is met, thereby obtaining a trained binocular depth estimation network. Exemplarily, the model training end condition can be that the first model loss value meets a first preset condition, such as the first model loss value is less than or equal to a threshold; or, the model training end condition can be that the number of model training iterations reaches a preset number.

[0110] It can be seen that a trained binocular depth estimation network can be obtained through the above steps 301 to 303.

[0111] In step 202, a coarse-grained completion operation is performed on the residual point cloud to be processed based on semantic information, and the point cloud after the coarse-grained completion operation is projected into a two-dimensional space to obtain a second depth image corresponding to the residual point cloud to be processed.

[0112] Exemplarily, after obtaining the semantic information of the target object in the binocular image to be processed, the semantic information can be used to perform a coarse-grained completion operation on the residual point cloud to be processed, and the point cloud after the coarse-grained completion operation is projected into a 2D space to obtain a second depth image corresponding to the residual point cloud.

[0113] In some embodiments, the optional implementation method of performing a coarse-grained completion operation on the residual dot cloud to be processed based on semantic information may include the following steps: based on the semantic information, obtaining a candidate point cloud for point cloud completion from the residual dot cloud to be processed, and performing a coarse-grained completion operation on the hole area in the residual dot cloud to be processed based on the candidate point cloud. In some embodiments, the optional implementation method of obtaining a candidate point cloud for point cloud completion from the residual dot cloud to be processed based on semantic information may include the following steps: based on the semantic information, determining the bounding rectangle of the target object from the residual dot cloud to be processed, obtaining the point cloud within the bounding rectangle from the residual dot cloud to be processed, and determining the hole area in the point cloud within the bounding rectangle, and determining the point cloud within the bounding rectangle that is around the hole area and in the same direction as the target object as the candidate point cloud for point cloud completion.

[0114] For example, Figure 4As shown, taking the weld in the workpiece as an example, the weld circumscribed rectangle can be determined (or calculated) from the residual defect cloud to be processed according to the semantic information of the weld. Then, the point cloud within the circumscribed rectangle is obtained from the residual defect cloud to be processed. Since the pixel values ​​in the hollow area of ​​the weld are inconsistent with the pixel values ​​in other areas of the weld, and there is a clear difference, the hollow area (or hollow edge) in the point cloud within the circumscribed rectangle can be determined based on the pixel values ​​in the first depth image, and the point cloud around the hollow area and in the same direction as the weld (or weld bead) can be intercepted in the circumscribed rectangle, and the intercepted point cloud is determined as a candidate point cloud for point cloud completion. Based on the candidate point cloud, the ICP (Iterative Closest Point) algorithm is used to roughly complete the point cloud of the hollow area to obtain the point cloud after rough completion.

[0115] In step 203, a fusion operation is performed on the first depth image and the second depth image to obtain a depth completion image.

[0116] In some embodiments, a fusion operation can be performed on the first depth image of the binocular image to be processed and the second depth image corresponding to the residual dot cloud based on a pre-trained deep fusion network to obtain a depth completion image. Exemplarily, the first depth image and the second depth image can be input into a pre-trained deep fusion network for fusion operation to obtain a depth completion image output by the deep fusion network. In some embodiments, the input of the deep fusion network includes the first depth image of the binocular image to be processed and the second depth image corresponding to the residual dot cloud, and the output of the deep fusion network is the depth completion image, and the deep fusion network has learned the mapping relationship between the first depth image of the binocular image and the second depth image corresponding to the residual dot cloud and the depth completion image.

[0117] In some embodiments, the above-mentioned deep fusion network can be pre-trained, such as Figure 5 As shown, the training method of the deep fusion network may include but is not limited to the following steps.

[0118] In step 501, a first depth image of a binocular image sample and a second depth image corresponding to a residual cloud sample are input into a depth fusion network for fusion operation to obtain a first depth completion image.

[0119] In some embodiments, the above-mentioned “fusion operation” may be an image superposition process, such as directly adding two depth images, but is not limited thereto. Exemplarily, the formula of the depth fusion network may be: depth_fusion= depth(1)+ depth(2), where depth(1) is the first depth image, is the weight corresponding to the first depth image during the fusion operation; depth(2) is the second depth image, is the weight corresponding to the second depth image during the fusion operation. , are the learnable parameters of the deep fusion network.

[0120] In some embodiments, the residual spot cloud sample may be a point cloud including a target object sample, and the target object sample in the binocular image sample and the target object sample in the residual spot cloud sample may be the same target object.

[0121] In step 502, the first depth completion image is back-projected to obtain a completed three-dimensional point cloud containing target object samples.

[0122] Exemplarily, the first depth completion image may be mapped from a 2D space to a 3D space to obtain a completed three-dimensional point cloud containing a sample of the target object.

[0123] In step 503, a second model loss value is determined based on the completed three-dimensional point cloud containing the target object sample, the complete point cloud associated with the residual point cloud sample, and the target object sample label corresponding to the point cloud.

[0124] Exemplarily, the second model loss value of the deep fusion network can be determined based on the completed three-dimensional point cloud containing the target object sample, the complete point cloud associated with the residual point cloud sample, and the target object sample label corresponding to the point cloud, combined with the model loss function of the deep fusion network. Exemplarily, the model loss function of the deep fusion network can be a cross entropy function, but is not limited thereto. For example, the model loss function of the deep fusion network can be an L1Loss loss function (also called mean absolute error).

[0125] In step 504, model training is performed on the deep fusion network based on the second model loss value until the model training end condition is met.

[0126] Exemplarily, a gradient descent algorithm can be used to perform gradient descent processing on the second model loss value, and the parameters in the deep fusion network can be updated according to the result of the gradient descent processing until the model training end condition is met, thereby obtaining a trained deep fusion network. Exemplarily, the model training end condition can be that the second model loss value meets a second preset condition, such as the second model loss value is less than or equal to a threshold; or, the model training end condition can be that the number of model training iterations reaches a preset number.

[0127] It can be seen that a trained deep fusion network can be obtained through the above steps 501 to 504.

[0128] In step 204, the depth completion image is back-projected to obtain a completed three-dimensional point cloud containing the target object.

[0129] Optional implementations of step 204 can be found in Figure 1A Optional implementations of step 104, and Figure 1A Other related parts in the embodiments involved will not be described in detail here.

[0130] In the above embodiment, the present disclosure introduces a point cloud completion preprocessing method combined with semantic information by introducing a binocular depth estimation network. With the help of semantic information, the coarse-grained completion of the point cloud is completed, which can further improve the convergence and completion effect of the subsequent model. In addition, the present disclosure provides a loss function structure based on semantic information for neural network training of point cloud completion. By combining the loss function of global and local consistency, the accuracy and robustness of point cloud completion can be further improved.

[0131] Figure 6 A flowchart of a point cloud completion model training method provided in an embodiment of the present disclosure. Figure 7 This is an example flow chart of the point cloud completion model training method provided in the embodiment of the present disclosure. In some embodiments, the point cloud completion model may include a binocular depth estimation network and a depth fusion network. Figure 6 and Figure 7 As shown, the point cloud completion model training method may include but is not limited to the following steps.

[0132] In step 601, a binocular image sample is input into a binocular depth estimation network for semantic analysis and depth information extraction to obtain semantic information of a target object sample in the binocular image sample and a first depth image of the binocular image sample.

[0133] For example, taking the target object sample as a weld in a workpiece as an example, a binocular image sample is obtained, and the binocular image sample includes a first image (i.e., Figure 7 RGB1 as shown) and the second image (i.e. Figure 7 RGB1 and RGB2 are input into the binocular depth estimation network for semantic analysis and depth information extraction to obtain the semantic information of the weld in the binocular image sample and the first depth image of the binocular image sample.

[0134] In step 602, a coarse-grained completion operation is performed on the residual point cloud sample based on the semantic information, and the point cloud after the coarse-grained completion operation is projected into a two-dimensional space to obtain a second depth image corresponding to the residual point cloud sample.

[0135] Exemplarily, a residual defect cloud sample is obtained, where the residual defect cloud sample is a point cloud including a target object sample, and the target object sample in the binocular image sample and the target object sample in the residual defect cloud sample are the same target object. Figure 7 As shown, semantic information is used to perform point cloud completion preprocessing (i.e., point cloud coarse-grained completion operation) on the hole area in the residual defect cloud sample, and the point cloud after point cloud completion preprocessing is projected into 2D space to obtain a second depth image corresponding to the residual defect cloud sample.

[0136] In step 603, a first model loss value is determined based on the semantic information, the first depth image, and the second depth image in combination with a model loss function adopted by the binocular depth estimation network.

[0137] For example, Figure 7 As shown, the depth mutual supervision is to train the binocular depth estimation network. The depth mutual supervision is performed through the model loss function adopted by the binocular depth estimation network. Exemplarily, the depth mutual supervision can use the model loss function adopted by the binocular depth estimation network to combine semantic information, the first depth image and the second depth image to determine the first model loss value. For optional implementation methods, see Figure 3 Optional implementations of step 302, and Figure 3 Other related parts in the embodiments involved will not be described in detail here.

[0138] In step 604, the first depth image and the second depth image are input into a depth fusion network for fusion operation to obtain a first depth completion image.

[0139] The optional implementation of step 604 can be found in the above Figure 5 Optional implementations of step 501, and Figure 5 Other related parts in the embodiments involved will not be described in detail here.

[0140] In step 605, the first depth completion image is back-projected to obtain a completed three-dimensional point cloud containing target object samples.

[0141] The optional implementation of step 605 can be found in the above Figure 5 Optional implementations of step 502, and Figure 5 Other related parts in the embodiments involved will not be described in detail here.

[0142] In step 606, a second model loss value is determined based on the completed three-dimensional point cloud containing the target object sample, the complete point cloud associated with the residual point cloud sample, and the target object sample label corresponding to the point cloud.

[0143] For example, take the target object sample as a weld of a workpiece as an example. Figure 7 As shown, after obtaining the completed three-dimensional point cloud containing the weld, one-to-one supervision of the point cloud can be performed based on the completed three-dimensional point cloud, the complete point cloud associated with the residual point cloud sample, and the weld label corresponding to the point cloud, and the loss value of the second model of the deep fusion network can be determined in combination with the loss function.

[0144] The optional implementation of step 606 can be found in the above Figure 5 Optional implementations of step 503, and Figure 5 Other related parts in the embodiments involved will not be described in detail here.

[0145] In step 607, the binocular depth estimation network is model trained based on the first model loss value, and the depth fusion network is model trained based on the second model loss value.

[0146] In some embodiments, the binocular depth estimation network and the deep fusion network can be trained together, or the binocular depth estimation network and the deep fusion network can also be trained separately, and the present disclosure does not specifically limit this. Exemplarily, when the first model loss value of the binocular depth estimation network is obtained, the binocular depth estimation network can be trained based on the first model loss value. After completing the training of the binocular depth estimation network, the parameters of the binocular depth estimation network are fixed, and the model training of the deep fusion network is performed.

[0147] In the above embodiment, the present disclosure introduces a point cloud completion preprocessing method combined with semantic information by introducing a binocular depth estimation network. With the help of semantic information, the coarse-grained completion of the point cloud is completed, which can further improve the convergence and completion effect of the subsequent model. In addition, the present disclosure provides a loss function structure based on semantic information for neural network training of point cloud completion. By combining the loss function of global and local consistency, the accuracy and robustness of point cloud completion can be further improved.

[0148] Figure 8 FIG. 1 is a flow chart of an end-to-end point cloud completion method provided by an embodiment of the present disclosure. Figure 8 As shown, the end-to-end point cloud completion method may include but is not limited to the following steps.

[0149] In step 801, a binocular image to be processed and a residual defect cloud to be processed are obtained.

[0150] In some embodiments, the residual point cloud to be processed may be a point cloud including a target object. The target object in the binocular image to be processed and the target object in the residual point cloud to be processed may be the same target object.

[0151] In step 802, the binocular image to be processed and the residual point cloud to be processed are input into a point cloud completion model to obtain a completed three-dimensional point cloud containing the target object.

[0152] In some embodiments, the point cloud completion model can be obtained by pre-training. The training method of the point cloud completion model can refer to the relevant description of any embodiment of the point cloud completion model training method described above, and will not be repeated here.

[0153] In the above embodiments, end-to-end point cloud completion can be achieved, complete and accurate point cloud completion can be achieved, the accuracy of point cloud completion can be significantly improved, and the point cloud completion effect can be improved. For example, it can be applied to weld seam recognition scenarios, which can improve the weld seam recognition effect, thereby improving the generalization ability, welding efficiency and quality of welding robots.

[0154] Fig. 9 This is a block diagram of a point cloud completion device based on semantic information provided by an embodiment of the present disclosure. Fig. 9 As shown, the point cloud completion device based on semantic information may include: a semantic analysis module 901, a completion preprocessing module 902, a fusion module 903 and a back-projection module 904.

[0155] The semantic analysis module 901 is used to perform image processing on the binocular image to be processed to obtain semantic information of the target object in the binocular image to be processed and a first depth image of the binocular image to be processed.

[0156] In some embodiments, the semantic analysis module 901 is used to: perform semantic analysis and depth information extraction on the binocular image to be processed based on a pre-trained binocular depth estimation network, and obtain semantic information of the target object in the binocular image to be processed and a first depth image of the binocular image to be processed; wherein the input of the binocular depth estimation network is the binocular image to be processed, the output of the binocular depth estimation network includes the semantic information of the target object and the first depth image, and the binocular depth estimation network has learned the mapping relationship between the binocular image and the semantic information of the target object and the first depth image, respectively.

[0157] In some embodiments, the binocular depth estimation network is trained in the following manner: the binocular image samples are input into the binocular depth estimation network for semantic analysis and depth information extraction to obtain the semantic information of the target object samples in the binocular image samples and the first depth image of the binocular image samples; based on the semantic information of the target object samples, the first depth image of the binocular image samples and the second depth image corresponding to the residual dot cloud samples, combined with the model loss function adopted by the binocular depth estimation network, a first model loss value is determined; the residual dot cloud samples are point clouds including the target object samples, and the target object samples in the binocular image samples and the target object samples in the residual dot cloud samples are the same target object; and the binocular depth estimation network is model trained based on the first model loss value.

[0158] In some embodiments, the model loss function adopted by the binocular depth estimation network is based on a first loss function, a second loss function and a third loss function; wherein the first loss function is used to constrain the scale of the first depth image to be consistent with the scale of the second depth image; the second loss function is combined with semantic information to constrain the geometric structure consistency of the same semantic pixels in the first depth image and the second depth image in the local space; the third loss function is used to constrain the first depth image and the second depth image to be consistent in global geometric structure.

[0159] The completion preprocessing module 902 is used to perform a coarse-grained completion operation on the residual point cloud to be processed based on semantic information, and project the point cloud after the coarse-grained completion operation into a two-dimensional space to obtain a second depth image corresponding to the residual point cloud. The residual point cloud to be processed is a point cloud including a target object, and the target object in the binocular image to be processed and the target object in the residual point cloud to be processed are the same target object.

[0160] In some embodiments, the completion preprocessing module 902 is used to: obtain candidate point clouds for point cloud completion from the residual point cloud to be processed based on semantic information; perform coarse-grained completion operations on the hole areas in the residual point cloud to be processed based on the candidate point clouds. In some embodiments, the completion preprocessing module 902 is used to: determine the bounding rectangle of the target object from the residual point cloud to be processed based on semantic information; obtain the point cloud within the bounding rectangle from the residual point cloud to be processed, and determine the hole area in the point cloud within the bounding rectangle; determine the point cloud within the bounding rectangle that is around the hole area and in the same direction as the target object as the candidate point cloud for point cloud completion.

[0161] The fusion module 903 is used to perform a fusion operation on the first depth image and the second depth image to obtain a depth completion image. In some embodiments, the fusion module 903 is used to: perform a fusion operation on the first depth image of the binocular image to be processed and the second depth image corresponding to the residual dot cloud based on a pre-trained deep fusion network to obtain a depth completion image; wherein the input of the deep fusion network includes the first depth image of the binocular image to be processed and the second depth image corresponding to the residual dot cloud, the output of the deep fusion network is the depth completion image, and the deep fusion network has learned the mapping relationship between the first depth image of the binocular image and the second depth image corresponding to the residual dot cloud and the depth completion image.

[0162] In some embodiments, the deep fusion network is trained in the following manner: a first depth image of a binocular image sample and a second depth image corresponding to a residual defect cloud sample are input into the deep fusion network for fusion operation to obtain a first depth completed image; the residual defect cloud sample is a point cloud including a target object sample, and the target object sample in the binocular image sample and the target object sample in the residual defect cloud sample are the same target object; the first depth completed image is back-projected to obtain a completed three-dimensional point cloud containing the target object sample; a second model loss value is determined based on the completed three-dimensional point cloud containing the target object sample, the complete point cloud associated with the residual defect cloud sample, and the target object sample label corresponding to the point cloud; and the deep fusion network is model trained based on the second model loss value.

[0163] The back-projection module 904 is used to back-project the depth completion image to obtain a completed three-dimensional point cloud containing the target object.

[0164] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0165] Fig.10 A block diagram of a point cloud completion model training device provided in an embodiment of the present disclosure. In some embodiments, the point cloud completion model may include but is not limited to a binocular depth estimation network and a depth fusion network. Fig.10 As shown, the point cloud completion model training device may include: a first acquisition module 1001, a completion preprocessing module 1002, a first determination module 1003, a fusion module 1004, a back projection module 1005, a second determination module 1006 and a training module 1007.

[0166] Among them, the first acquisition module 1001 is used to input the binocular image samples into the binocular depth estimation network for semantic analysis and depth information extraction, so as to obtain the semantic information of the target object samples in the binocular image samples and the first depth image of the binocular image samples.

[0167] The completion preprocessing module 1002 is used to perform a coarse-grained completion operation on the residual defect cloud sample based on the semantic information, and project the point cloud after the coarse-grained completion operation into a two-dimensional space to obtain a second depth image corresponding to the residual defect cloud sample. The residual defect cloud sample is a point cloud including the target object sample, and the target object sample in the binocular image sample and the target object sample in the residual defect cloud sample are the same target object.

[0168] The first determination module 1003 is used to determine a first model loss value based on the semantic information, the first depth image and the second depth image in combination with a model loss function adopted by the binocular depth estimation network.

[0169] In some embodiments, the model loss function adopted by the binocular depth estimation network is based on a first loss function, a second loss function and a third loss function; wherein the first loss function is used to constrain the scale of the first depth image to be consistent with the scale of the second depth image; the second loss function is combined with semantic information to constrain the geometric structure consistency of the same semantic pixels in the first depth image and the second depth image in the local space; the third loss function is used to constrain the first depth image and the second depth image to be consistent in global geometric structure.

[0170] The fusion module 1004 is used to input the first depth image and the second depth image into the depth fusion network for fusion operation to obtain a first depth completion image.

[0171] The back-projection module 1005 is used to back-project the first depth completion image to obtain a completed three-dimensional point cloud containing the target object sample.

[0172] The second determination module 1006 is used to determine a second model loss value based on the completed three-dimensional point cloud containing the target object sample, the complete point cloud associated with the residual point cloud sample, and the target object sample label corresponding to the point cloud.

[0173] The training module 1007 is used to perform model training on the binocular depth estimation network based on the first model loss value, and to perform model training on the depth fusion network based on the second model loss value.

[0174] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0175] Fig.11 Schematic diagram of an end-to-end point cloud completion device provided by an embodiment of the present disclosure. Fig.11As shown, the end-to-end point cloud completion device may include: a first acquisition module 1101 and a second acquisition module 1102.

[0176] Among them, the first acquisition module 1101 is used to obtain the binocular image to be processed and the residual point cloud to be processed. The residual point cloud to be processed is a point cloud including the target object. The target object in the binocular image to be processed and the target object in the residual point cloud to be processed are the same target object.

[0177] The second acquisition module 1102 is used to input the binocular image to be processed and the residual point cloud to be processed into the point cloud completion model to obtain a completed three-dimensional point cloud containing the target object.

[0178] In some embodiments, the point cloud completion model can be obtained by pre-training. The training method of the point cloud completion model can refer to the relevant description of any embodiment of the point cloud completion model training method described above, and will not be repeated here.

[0179] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0180] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0181] like Fig.12 , is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0182] like Fig.12As shown, the electronic device includes: one or more processors 1201, a memory 1202, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.12 A processor 1201 is taken as an example.

[0183] The memory 1202 is a non-transitory computer-readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the method described in any of the above embodiments provided by the present disclosure. The non-transitory computer-readable storage medium of the present disclosure stores computer instructions, which are used to enable a computer to perform the method described in any of the above embodiments provided by the present disclosure.

[0184] The memory 1202 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the point cloud completion method based on semantic information, the point cloud completion model training method, and the end-to-end point cloud completion method in the embodiments of the present disclosure. The processor 1201 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1202, that is, the point cloud completion method based on semantic information, the point cloud completion model training method, and the end-to-end point cloud completion method in the above method embodiments are implemented.

[0185] The memory 1202 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1202 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1202 may optionally include a memory remotely arranged relative to the processor 1201, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0186] The electronic device may further include: an input device 1203 and an output device 1204. The processor 1201, the memory 1202, the input device 1203 and the output device 1204 may be connected via a bus or other means. Fig.12 The example of connecting through bus is taken in the following.

[0187] The input device 1203 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, and other input devices. The output device 1204 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0188] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0189] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0190] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0191] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0192] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0193] In the description of the present disclosure, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0194] In addition, in the description of the present disclosure, “plurality” means at least two, for example, two, three, etc., unless otherwise clearly and specifically defined.

[0195] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0196] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0197] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0198] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0199] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0200] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A point cloud completion method based on semantic information, characterized in that: include: Performing image processing on the binocular image to be processed to obtain semantic information of the target object in the binocular image to be processed and a first depth image of the binocular image to be processed; Performing a coarse-grained completion operation on the residual point cloud to be processed based on the semantic information, and projecting the point cloud after the coarse-grained completion operation into a two-dimensional space to obtain a second depth image corresponding to the residual point cloud to be processed; The residual point cloud to be processed is a point cloud including the target object, and the target object in the binocular image to be processed and the target object in the residual point cloud to be processed are the same target object; Performing a fusion operation on the first depth image and the second depth image to obtain a depth completion image; Back-projecting the depth completion image to obtain a completed three-dimensional point cloud containing the target object.

2. The method according to claim 1, characterized in that The coarse-grained completion operation on the residual defect cloud to be processed based on the semantic information includes: Based on the semantic information, obtaining a candidate point cloud for point cloud completion from the residual point cloud to be processed; Based on the candidate point cloud, a coarse-grained filling operation is performed on the hole area in the residual point cloud to be processed.

3. The method according to claim 2, characterized in that The step of obtaining a candidate point cloud for point cloud completion from the residual point cloud to be processed based on the semantic information includes: Based on the semantic information, determining the circumscribed rectangle of the target object from the residual defect cloud to be processed; Acquire the point cloud within the circumscribed rectangle from the residual point cloud to be processed, and determine the hole area in the point cloud within the circumscribed rectangle; The point cloud within the circumscribed rectangle, which is around the hole area and in the same direction as the target object, is determined as the candidate point cloud for point cloud completion.

4. The method according to claim 1, characterized in that The performing image processing on the binocular image to be processed to obtain semantic information of the target object in the binocular image to be processed and a first depth image of the binocular image to be processed includes: Based on a pre-trained binocular depth estimation network, semantic analysis and depth information extraction are performed on the binocular image to be processed to obtain semantic information of the target object in the binocular image to be processed and a first depth image of the binocular image to be processed; Among them, the input of the binocular depth estimation network is the binocular image to be processed, the output of the binocular depth estimation network includes semantic information of the target object and the first depth image, and the binocular depth estimation network has learned the mapping relationship between the binocular image and the semantic information of the target object and the first depth image respectively.

5. The method according to claim 4, characterized in that The binocular depth estimation network is trained in the following way: Inputting the binocular image sample into the binocular depth estimation network for semantic analysis and depth information extraction to obtain semantic information of the target object sample in the binocular image sample and a first depth image of the binocular image sample; Based on the semantic information of the target object sample, the first depth image of the binocular image sample and the second depth image corresponding to the residual defect cloud sample, in combination with the model loss function adopted by the binocular depth estimation network, a first model loss value is determined; the residual defect cloud sample is a point cloud including the target object sample, and the target object sample in the binocular image sample and the target object sample in the residual defect cloud sample are the same target object; Model training is performed on the binocular depth estimation network based on the first model loss value.

6. The method according to claim 5, characterized in that The model loss function adopted by the binocular depth estimation network is based on a first loss function, a second loss function and a third loss function; wherein, The first loss function is used to constrain the scale of the first depth image and the scale of the second depth image to be consistent; The second loss function is combined with the semantic information to constrain the geometric structure consistency of the same semantic pixel points in the first depth image and the second depth image in the local space; The third loss function is used to constrain the first depth image and the second depth image to maintain consistency in global geometric structure.

7. The method according to any one of claims 1 to 6, characterized in that The fusing operation of the first depth image and the second depth image to obtain a depth completion image includes: Based on a pre-trained depth fusion network, a first depth image of the binocular image to be processed and a second depth image corresponding to the residual defect cloud are fused to obtain the depth completion image; Among them, the input of the deep fusion network includes the first depth image of the binocular image to be processed and the second depth image corresponding to the residual spot cloud, the output of the deep fusion network is the depth completion image, and the deep fusion network has learned the mapping relationship between the first depth image of the binocular image and the second depth image corresponding to the residual spot cloud and the depth completion image.

8. The method according to claim 7, characterized in that The deep fusion network is trained in the following way: Inputting the first depth image of the binocular image sample and the second depth image corresponding to the residual defect cloud sample into the depth fusion network for fusion operation to obtain a first depth completion image; The residual defect cloud sample is a point cloud including a target object sample, and the target object sample in the binocular image sample and the target object sample in the residual defect cloud sample are the same target object; Back-projecting the first depth completion image to obtain a completed three-dimensional point cloud containing the target object sample; Determine a second model loss value based on a completed three-dimensional point cloud containing the target object sample, a complete point cloud associated with the residual point cloud sample, and a target object sample label corresponding to the point cloud; The deep fusion network is trained based on the second model loss value.

9. A point cloud completion model training method, characterized in that: The point cloud completion model includes a binocular depth estimation network and a depth fusion network, and the training method includes: Inputting the binocular image sample into the binocular depth estimation network for semantic analysis and depth information extraction to obtain semantic information of the target object sample in the binocular image sample and a first depth image of the binocular image sample; Based on the semantic information, a coarse-grained completion operation is performed on the residual defect cloud sample, and the point cloud after the coarse-grained completion operation is projected into a two-dimensional space to obtain a second depth image corresponding to the residual defect cloud sample; the residual defect cloud sample is a point cloud including the target object sample, and the target object sample in the binocular image sample and the target object sample in the residual defect cloud sample are the same target object; Determine a first model loss value based on the semantic information, the first depth image, and the second depth image in combination with a model loss function adopted by the binocular depth estimation network; Inputting the first depth image and the second depth image into the depth fusion network for fusion operation to obtain a first depth completion image; Back-projecting the first depth completion image to obtain a completed three-dimensional point cloud containing the target object sample; Determine a second model loss value based on a completed three-dimensional point cloud containing the target object sample, a complete point cloud associated with the residual point cloud sample, and a target object sample label corresponding to the point cloud; The binocular depth estimation network is model trained based on the first model loss value, and the depth fusion network is model trained based on the second model loss value.

10. The method according to claim 9, characterized in that The model loss function adopted by the binocular depth estimation network is based on a first loss function, a second loss function and a third loss function; wherein, The first loss function is used to constrain the scale of the first depth image and the scale of the second depth image to be consistent; The second loss function is combined with the semantic information to constrain the geometric structure consistency of the same semantic pixel points in the first depth image and the second depth image in the local space; The third loss function is used to constrain the first depth image and the second depth image to maintain consistency in global geometric structure.

11. An end-to-end point cloud completion method, characterized in that: include: Acquire a binocular image to be processed and a residual spot cloud to be processed, wherein the residual spot cloud to be processed is a point cloud including a target object, and the target object in the binocular image to be processed and the target object in the residual spot cloud to be processed are the same target object; The binocular image to be processed and the residual point cloud to be processed are input into a point cloud completion model to obtain a completed three-dimensional point cloud containing the target object; wherein the point cloud completion model is a model trained based on the method as described in claim 9 or 10.

12. A point cloud completion device based on semantic information, characterized in that: include: A semantic analysis module, used for performing image processing on the binocular image to be processed to obtain semantic information of the target object in the binocular image to be processed and a first depth image of the binocular image to be processed; A completion preprocessing module, configured to perform a coarse-grained completion operation on the residual point cloud to be processed based on the semantic information, and project the point cloud after the coarse-grained completion operation into a two-dimensional space to obtain a second depth image corresponding to the residual point cloud; the residual point cloud to be processed is a point cloud including the target object, and the target object in the binocular image to be processed and the target object in the residual point cloud to be processed are the same target object; A fusion module, configured to perform a fusion operation on the first depth image and the second depth image to obtain a depth completion image; The back-projection module is used to back-project the depth completion image to obtain a completed three-dimensional point cloud containing the target object.

13. A point cloud completion model training device, characterized in that: The point cloud completion model includes a binocular depth estimation network and a depth fusion network, and the training device includes: A first acquisition module is used to input the binocular image sample into the binocular depth estimation network for semantic analysis and depth information extraction, so as to obtain the semantic information of the target object sample in the binocular image sample and the first depth image of the binocular image sample; A completion preprocessing module, configured to perform a coarse-grained completion operation on the residual defect cloud sample based on the semantic information, and project the point cloud after the coarse-grained completion operation into a two-dimensional space to obtain a second depth image corresponding to the residual defect cloud sample; the residual defect cloud sample is a point cloud including the target object sample, and the target object sample in the binocular image sample and the target object sample in the residual defect cloud sample are the same target object; A first determination module, configured to determine a first model loss value based on the semantic information, the first depth image, and the second depth image in combination with a model loss function adopted by the binocular depth estimation network; A fusion module, used for inputting the first depth image and the second depth image into the depth fusion network for fusion operation to obtain a first depth completion image; A back-projection module, configured to back-project the first depth completion image to obtain a completed three-dimensional point cloud containing a sample of the target object; A second determination module is used to determine a second model loss value based on the completed three-dimensional point cloud containing the target object sample, the complete point cloud associated with the residual point cloud sample, and the target object sample label corresponding to the point cloud; A training module is used to perform model training on the binocular depth estimation network based on the first model loss value, and to perform model training on the depth fusion network based on the second model loss value.

14. An end-to-end point cloud completion device, characterized in that: include: A first acquisition module is used to acquire a binocular image to be processed and a residual spot cloud to be processed, wherein the residual spot cloud to be processed is a point cloud including a target object, and the target object in the binocular image to be processed and the target object in the residual spot cloud to be processed are the same target object; The second acquisition module is used to input the binocular image to be processed and the residual point cloud to be processed into a point cloud completion model to obtain a completed three-dimensional point cloud containing the target object; wherein the point cloud completion model is a model trained based on the method as described in claim 9 or 10.

15. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8, 9-10, and 11.

16. A storage medium storing instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1-8, 9-10 and 11.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8, 9 to 10 and 11.

Images