General 3D bounding box generation method and device based on geometric structure

Abstract

The invention provides a general 3D bounding box generation method and device based on a geometric structure. The method comprises the following steps: generating initial 3D bounding box data based on a pre-trained instance segmentation detector and 2D bounding box data; and inputting the initial 3D bounding box data into a pre-trained 3D bounding box generation model, and obtaining target 3D bounding box data generated by the 3D bounding box generation model. According to the general 3D bounding box generation method and device based on the geometric structure provided by the invention, the initial 3D bounding box data generated according to the 2D bounding box data is input into the pre-trained 3D bounding box generation model, so that the generated target 3D bounding box data can be obtained, and the 3D bounding box can be generated through the length-width ratio semantic loss function and the boundary projection loss function. The point cloud and the bounding box align the loss function to supervise the information loss problem caused by a common 3D bounding box model, and the accuracy of the target 3D bounding box data can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for generating a general 3D bounding box based on a geometric structure. Background Art

[0002] 3D object detection is an important part of autonomous driving tasks. It takes sensor data (such as LiDAR point cloud, RGB image, etc.) as input and outputs the 3D bounding box of the corresponding target. The current fully supervised 3D object detector requires a large number of annotated 3D bounding boxes for training. Because 3D bounding box annotation needs to consider multiple attributes (center point position, length, width, height and rotation angle) and alignment under different perspectives, 3D bounding box annotation requires a lot of manpower and material resources.

[0003] Compared with the annotation of 3D object detection boxes, the annotation of 2D object detection boxes can save 3-16 times of time. Therefore, some works hope to assist in the generation of 3D object detection boxes by using 2D object detection boxes plus corresponding LiDAR point clouds. Since 2D bounding boxes lose a lot of spatial information (such as depth information and rotation angle) compared to 3D bounding boxes, existing works often need to use strong semantic prior information, such as CAD models, to assist in model learning. This strong prior information limits the extension of these methods to new categories (such as cycling, people, tables, chairs, etc.) and new scenes (such as indoors and outdoors). Summary of the invention

[0004] The present invention provides a general 3D bounding box generation method and device based on geometric structure, which are used to solve the technical problem of low accuracy of 3D bounding box caused by loss of spatial information when generating 3D bounding box from 2D bounding box in the prior art.

[0005] The present invention provides a general 3D bounding box generation method based on geometric structure, comprising the following steps.

[0006] Generate initial 3D bounding box data based on the pre-trained instance segmentation detector and 2D bounding box data; The initial 3D bounding box data is input into a pre-trained 3D bounding box generation model, and target 3D bounding box data generated by the 3D bounding box generation model is obtained, wherein the 3D bounding box generation model is trained based on a boundary projection loss function, an aspect ratio semantic loss function, and a point cloud and bounding box alignment loss function, and the accuracy of the target 3D bounding box data is greater than that of the initial 3D bounding box data.

[0007] In some embodiments, the training step of generating a 3D bounding box model includes: Determining a confidence score based on the initial 3D bounding box data, the confidence score being used to represent the accuracy of the initial 3D bounding box data; The 3D bounding box generation model is trained based on the boundary projection loss function, the aspect ratio semantic loss function, the point cloud and bounding box alignment loss function and the confidence score.

[0008] In some embodiments, the training of the 3D bounding box generation model based on the boundary projection loss function, the aspect ratio semantic loss function, the point cloud and bounding box alignment loss function and the confidence score comprises: Determining a center point of the target 3D bounding box based on the confidence score; Determining the range of the target 3D bounding box in 3D space based on the boundary projection loss function; Determining an aspect ratio of the target 3D bounding box based on the aspect ratio semantic loss function; The rotation angle and side length of the target 3D bounding box are determined based on the point cloud and bounding box alignment loss function.

[0009] In some embodiments, determining a confidence score based on the initial 3D bounding box data comprises: Determining a heat map and a centrality based on the initial 3D bounding box data; The confidence score is determined based on the heat map and the centrality.

[0010] In some embodiments, the method further comprises: Obtain aspect ratio information based on language macromodel.

[0011] In some embodiments, the initial 3D bounding box data includes indoor point cloud data and / or outdoor point cloud data.

[0012] In some embodiments, when the initial 3D bounding box data includes indoor point cloud data, the backbone network of the 3D bounding box generation model is FCAF3D.

[0013] In some embodiments, when the initial 3D bounding box data includes outdoor point cloud data, the backbone network of the 3D bounding box generation model is Centerpoint.

[0014] The present invention also provides a general 3D bounding box generation device based on geometric structure, comprising the following modules: A first generation module, for generating initial 3D bounding box data based on a pre-trained instance segmentation detector and 2D bounding box data; The first acquisition module is used to input the initial 3D bounding box data into a pre-trained 3D bounding box generation model to obtain target 3D bounding box data generated by the 3D bounding box generation model, wherein the 3D bounding box generation model is trained based on a boundary projection loss function, an aspect ratio semantic loss function, and a point cloud and bounding box alignment loss function, and the accuracy of the target 3D bounding box data is greater than that of the initial 3D bounding box data.

[0015] In some embodiments, the geometric structure-based general 3D bounding box generation apparatus further includes a first training module, wherein the first training module includes: A first determination submodule, configured to determine a confidence score based on the initial 3D bounding box data, wherein the confidence score is used to represent the accuracy of the initial 3D bounding box data; The first training submodule is used to train the 3D bounding box generation model based on the boundary projection loss function, the aspect ratio semantic loss function, the point cloud and bounding box alignment loss function and the confidence score.

[0016] In some embodiments, the first training submodule includes: A first determining unit, configured to determine a center point of the target 3D bounding box based on the confidence score; A second determining unit, configured to determine a range of the target 3D bounding box in the 3D space based on the boundary projection loss function; A third determining unit, configured to determine the aspect ratio of the target 3D bounding box based on the aspect ratio semantic loss function; The fourth determining unit is used to determine the rotation angle and side length of the target 3D bounding box based on the point cloud and bounding box alignment loss function.

[0017] In some embodiments, the first determining submodule includes: a fifth determining unit, configured to determine a heat map and a centrality based on the initial 3D bounding box data; A sixth determination unit is used to determine the confidence score based on the heat map and the centrality.

[0018] In some embodiments, the general 3D bounding box generation apparatus based on geometric structure further includes: The second acquisition module is used to acquire aspect ratio information based on the language large model.

[0019] In some embodiments, the initial 3D bounding box data includes indoor point cloud data and / or outdoor point cloud data.

[0020] In some embodiments, when the initial 3D bounding box data includes indoor point cloud data, the backbone network of the 3D bounding box generation model is FCAF3D.

[0021] In some embodiments, when the initial 3D bounding box data includes outdoor point cloud data, the backbone network of the 3D bounding box generation model is Centerpoint.

[0022] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned general 3D bounding box generation methods based on geometric structures is implemented.

[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the general 3D bounding box generation method based on geometric structure as described above is implemented.

[0024] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned general 3D bounding box generation methods based on geometric structures.

[0025] The general 3D bounding box generation method and device based on geometric structure provided by the present invention can obtain the generated target 3D bounding box data by inputting the initial 3D bounding box data generated according to the 2D bounding box data into the pre-trained 3D bounding box generation model, and can supervise the information loss problem caused by the ordinary 3D bounding box model through the aspect ratio semantic loss function, the boundary projection loss function, and the point cloud and bounding box alignment loss function, so as to improve the accuracy of the target 3D bounding box data. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a flowchart of a general 3D bounding box generation method for geometric structures provided by the present invention.

[0028] Figure 2 It is a structural schematic diagram of a general 3D bounding box generation device based on geometric structure provided by the present invention.

[0029] Figure 3It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Figure 1 FIG. 1 is a flow chart of a general 3D bounding box generation method for a geometric structure provided by the present invention. Figure 1 As shown, the method includes the following: Step 101: Generate initial 3D bounding box data based on a pre-trained instance segmentation detector and 2D bounding box data.

[0032] Specifically, in order to solve the problem of "a lot of spatial information is missing in the 2D detection box" in the current 3D weakly supervised annotation method based on the assistance of the 2D target detection box, the present invention can first select the points projected into the corresponding 2D bounding box as the point cloud representing the corresponding target. Since the directly obtained point cloud contains noise (the point cloud of the surrounding targets), we use the pre-trained instance segmentation detector to denoise and finally obtain In-Box-Points, and then we generate the initial 3D bounding box data based on In-Box-Points.

[0033] Step 102: input the initial 3D bounding box data into a pre-trained 3D bounding box generation model, and obtain target 3D bounding box data generated by the 3D bounding box generation model, wherein the 3D bounding box generation model is trained based on a boundary projection loss function, an aspect ratio semantic loss function, and a point cloud and bounding box alignment loss function, and the accuracy of the target 3D bounding box data is greater than that of the initial 3D bounding box data.

[0034] Specifically, the initial 3D bounding box data is input into a pre-trained 3D bounding box generation model, which uses three different loss functions to deal with the information loss problem caused by the 2D bounding box to supervise the 3D bounding box model.

[0035] Among them, the boundary projection loss function is used to supervise the range of the target 3D bounding box in 3D space, the aspect ratio semantic loss function is used to supervise the aspect ratio of the target 3D bounding box, and the point cloud and bounding box alignment loss function is used to implicitly supervise the rotation angle and side length of the target 3D bounding box.

[0036] By supervising various properties of the 3D bounding box, the target 3D bounding box data can be obtained.

[0037] It is worth mentioning that this 3D bounding box generation model can also be used to train any 3D object detector.

[0038] The general 3D bounding box generation method based on geometric structure provided by the present invention can obtain the generated target 3D bounding box data by inputting the initial 3D bounding box data generated according to the 2D bounding box data into the pre-trained 3D bounding box generation model, and can supervise the information loss problem caused by the ordinary 3D bounding box model through the aspect ratio semantic loss function, the boundary projection loss function, and the point cloud and bounding box alignment loss function, so as to improve the accuracy of the target 3D bounding box data.

[0039] In some embodiments, the initial 3D bounding box data includes indoor point cloud data and / or outdoor point cloud data.

[0040] In some embodiments, when the initial 3D bounding box data includes indoor point cloud data, the backbone network of the 3D bounding box generation model is FCAF3D.

[0041] In some embodiments, when the initial 3D bounding box data includes outdoor point cloud data, the backbone network of the 3D bounding box generation model is Centerpoint.

[0042] Specifically, since indoor point cloud data is often very dense, the number of point clouds can be downsampled to 100,000 using the farthest point sampling for computational efficiency. For indoor and outdoor point clouds, the RANSAC method can be used to estimate the ground and remove it.

[0043] Indoor and outdoor 3D object detection research often adopts different backbone networks. For example, Centerpoint can be selected as the outdoor backbone network and FCAF3D as the indoor backbone network.

[0044] Then, for indoor and outdoor, the ProposalHead corresponding to Centerpoint and FCAF3D can be used respectively.

[0045] The general 3D bounding box generation method based on geometric structure provided by the present invention solves the problem that previous methods require strong semantic priors and are difficult to expand to new categories and scenes, and can be expanded to multiple indoor and outdoor categories.

[0046] In some embodiments, the training step of generating a 3D bounding box model includes: Determining a confidence score based on the initial 3D bounding box data, the confidence score being used to represent the accuracy of the initial 3D bounding box data; The 3D bounding box generation model is trained based on the boundary projection loss function, the aspect ratio semantic loss function, the point cloud and bounding box alignment loss function and the confidence score.

[0047] Specifically, for the training of the 3D bounding box generation model, center point supervision is first performed.

[0048] The initial 3D bounding box data can be used to determine the heatmap and centrality, and then the confidence scores of the indoor backbone network and the outdoor supervisor network are supervised respectively according to the heatmap and centrality.

[0049] It is worth mentioning that the confidence score is used to represent the probability that the point cloud is within the 3D bounding box.

[0050] Finally, the 3D bounding box generation model is trained through boundary projection loss function, aspect ratio semantic loss function, point cloud and bounding box alignment loss function and confidence score.

[0051] The general 3D bounding box generation method based on geometric structure provided by the present invention uses three different loss functions to cope with the information loss problem caused by supervising the 3D bounding box model with a 2D bounding box, and can supervise various attributes of the 3D bounding box to improve the accuracy of the target 3D bounding box data.

[0052] In some embodiments, determining a confidence score based on the initial 3D bounding box data comprises: Determining a heat map and a centrality based on the initial 3D bounding box data; The confidence score is determined based on the heat map and the centrality.

[0053] Specifically, a heatmap is a type of chart used to visualize data, which uses color coding to show the density and distribution of data points. It can intuitively display the areas and trends in a large number of data sets to help users discover patterns and regularities in the data.

[0054] Centerness is used to indicate how close a 3D bounding box is to the center, and can be used to exclude 3D bounding boxes that are far from the center.

[0055] The confidence score can be obtained by quantitatively scoring the heat map and centrality of the 3D bounding box data. The confidence score can be used to indicate whether the 3D bounding box data is accurate.

[0056] For example, for a certain 3D bounding box data, its confidence score is a according to its heat map and centrality, and if a is less than a preset confidence threshold c, it can be determined that the 3D bounding box data is inaccurate.

[0057] For another example, for a certain 3D bounding box data, its confidence score is b according to its heat map and centrality, and if b is greater than a preset confidence threshold c, it can be determined that the 3D bounding box data is accurate.

[0058] The general 3D bounding box generation method based on geometric structure provided by the present invention can supervise the confidence score through heat map and centrality, thereby supervising the center point of the 3D bounding box and improving the accuracy of the target 3D bounding box data.

[0059] In some embodiments, the training of the 3D bounding box generation model based on the boundary projection loss function, the aspect ratio semantic loss function, the point cloud and bounding box alignment loss function and the confidence score comprises: Determining a center point of the target 3D bounding box based on the confidence score; Determining the range of the target 3D bounding box in 3D space based on the boundary projection loss function; Determining an aspect ratio of the target 3D bounding box based on the aspect ratio semantic loss function; The rotation angle and side length of the target 3D bounding box are determined based on the point cloud and bounding box alignment loss function.

[0060] Specifically, the confidence score is supervised by heatmap and center point to supervise the center point of the 3D bounding box.

[0061] A boundary projection loss function is proposed to supervise the estimation of the range of the 3D bounding box in 3D space. Specifically, after the 3D bounding box is projected onto the image, the Manhattan distance is calculated with the 2D bounding box.

[0062]

[0063] in , , representing the positions of the four sides of the real 2D bounding box. According to the projection matrix Project the estimated target 3D bounding box onto the four sides of the corresponding 2D bounding box on the image. represents the boundary projection loss function.

[0064] The aspect ratio semantic loss function is proposed to supervise the learning of 3D object detection boxes. The shorter side of the estimated 3D bounding box is used as the width and the longer side as the length, and then the following supervision is performed:

[0065] in, represents the aspect ratio semantic loss function, r represents the target aspect ratio obtained from the language model, Represents the predicted 3D bounding box length and width.

[0066] The point cloud and bounding box alignment loss function is proposed to implicitly supervise the rotation angle and side length of the 3D bounding box. The distance from the point cloud to the four sides of the 3D bounding box can be calculated from the bird's eye view. . And the following loss function is used to constrain:

[0067] in, and represents the loss function and ReLU represents the activation function.

[0068] Furthermore, after the model is trained, it can directly generate the corresponding target 3D bounding box and can be used for the training of any 3D target detector.

[0069] The general 3D bounding box generation method based on geometric structure provided by the present invention can obtain the generated target 3D bounding box data by inputting the initial 3D bounding box data generated according to the 2D bounding box data into the pre-trained 3D bounding box generation model, and can supervise the information loss problem caused by the ordinary 3D bounding box model through the aspect ratio semantic loss function, the boundary projection loss function, and the point cloud and bounding box alignment loss function, so as to improve the accuracy of the target 3D bounding box data.

[0070] In some embodiments, the method further comprises: Obtain aspect ratio information based on language macromodel.

[0071] Specifically, in response to the problems of the current 3D weakly supervised labeling methods assisted by 2D object detection boxes, which "require strong semantic information prior and can only process fixed categories and scenes", the present invention is inspired by the recent large language models (such as ChatGPT, Wenxinyiyan). The large models contain a lot of human prior knowledge, from which we can obtain the geometric prior information of any target (3D bounding box aspect ratio information), so our method can be extended to most categories and scenes.

[0072] The language model can be used to obtain semantically rich aspect ratio information, providing supervision for the subsequent determination of the aspect ratio of the target 3D bounding box based on the aspect ratio semantic loss function.

[0073] The general 3D bounding box generation method based on geometric structure provided by the present invention can obtain a large amount of prior information by obtaining semantically rich aspect ratio information through a large language model, which can solve the problem that previous methods require strong semantic priors and are difficult to expand to new categories and scenes, thereby being able to be expanded to multiple indoor and outdoor categories.

[0074] Figure 2 is a schematic diagram of the structure of a general 3D bounding box generation device based on geometric structure provided by the present invention, such as Figure 2 As shown, the general 3D bounding box generation device based on geometric structure provided by the present invention includes a first generation module 201 and a first acquisition module 202, wherein: A first generating module 201, for generating initial 3D bounding box data based on a pre-trained instance segmentation detector and 2D bounding box data; The first acquisition module 202 is used to input the initial 3D bounding box data into a pre-trained 3D bounding box generation model to obtain target 3D bounding box data generated by the 3D bounding box generation model, wherein the 3D bounding box generation model is trained based on a boundary projection loss function, an aspect ratio semantic loss function, and a point cloud and bounding box alignment loss function, and the accuracy of the target 3D bounding box data is greater than that of the initial 3D bounding box data.

[0075] In some embodiments, the geometric structure-based general 3D bounding box generation apparatus further includes a first training module, wherein the first training module includes: A first determination submodule, configured to determine a confidence score based on the initial 3D bounding box data, wherein the confidence score is used to represent the accuracy of the initial 3D bounding box data; The first training submodule is used to train the 3D bounding box generation model based on the boundary projection loss function, the aspect ratio semantic loss function, the point cloud and bounding box alignment loss function and the confidence score.

[0076] In some embodiments, the first training submodule includes: A first determining unit, configured to determine a center point of the target 3D bounding box based on the confidence score; A second determining unit, configured to determine a range of the target 3D bounding box in the 3D space based on the boundary projection loss function; A third determining unit, configured to determine the aspect ratio of the target 3D bounding box based on the aspect ratio semantic loss function; The fourth determining unit is used to determine the rotation angle and side length of the target 3D bounding box based on the point cloud and bounding box alignment loss function.

[0077] In some embodiments, the first determining submodule includes: a fifth determining unit, configured to determine a heat map and a centrality based on the initial 3D bounding box data; A sixth determination unit is used to determine the confidence score based on the heat map and the centrality.

[0078] In some embodiments, the general 3D bounding box generation apparatus based on geometric structure further includes: The second acquisition module is used to acquire aspect ratio information based on the language large model.

[0079] In some embodiments, the initial 3D bounding box data includes indoor point cloud data and / or outdoor point cloud data.

[0080] In some embodiments, when the initial 3D bounding box data includes indoor point cloud data, the backbone network of the 3D bounding box generation model is FCAF3D.

[0081] In some embodiments, when the initial 3D bounding box data includes outdoor point cloud data, the backbone network of the 3D bounding box generation model is Centerpoint.

[0082] Specifically, the general 3D bounding box generation device based on geometric structure provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned general 3D bounding box generation method embodiment based on geometric structure, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0083] Figure 3 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute a general 3D bounding box generation method based on a geometric structure, the method comprising: Generate initial 3D bounding box data based on the pre-trained instance segmentation detector and 2D bounding box data; The initial 3D bounding box data is input into a pre-trained 3D bounding box generation model, and target 3D bounding box data generated by the 3D bounding box generation model is obtained, wherein the 3D bounding box generation model is trained based on a boundary projection loss function, an aspect ratio semantic loss function, and a point cloud and bounding box alignment loss function, and the accuracy of the target 3D bounding box data is greater than that of the initial 3D bounding box data.

[0084] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0085] In some embodiments, the training step of generating a 3D bounding box model includes: Determining a confidence score based on the initial 3D bounding box data, the confidence score being used to represent the accuracy of the initial 3D bounding box data; The 3D bounding box generation model is trained based on the boundary projection loss function, the aspect ratio semantic loss function, the point cloud and bounding box alignment loss function and the confidence score.

[0086] In some embodiments, the training of the 3D bounding box generation model based on the boundary projection loss function, the aspect ratio semantic loss function, the point cloud and bounding box alignment loss function and the confidence score comprises: Determining a center point of the target 3D bounding box based on the confidence score; Determining the range of the target 3D bounding box in 3D space based on the boundary projection loss function; Determining an aspect ratio of the target 3D bounding box based on the aspect ratio semantic loss function; The rotation angle and side length of the target 3D bounding box are determined based on the point cloud and bounding box alignment loss function.

[0087] In some embodiments, determining a confidence score based on the initial 3D bounding box data comprises: Determining a heat map and a centrality based on the initial 3D bounding box data; The confidence score is determined based on the heat map and the centrality.

[0088] In some embodiments, the method further comprises: Obtain aspect ratio information based on language macromodel.

[0089] In some embodiments, the initial 3D bounding box data includes indoor point cloud data and / or outdoor point cloud data.

[0090] In some embodiments, when the initial 3D bounding box data includes indoor point cloud data, the backbone network of the 3D bounding box generation model is FCAF3D.

[0091] In some embodiments, when the initial 3D bounding box data includes outdoor point cloud data, the backbone network of the 3D bounding box generation model is Centerpoint.

[0092] Specifically, the above-mentioned electronic device provided in the embodiment of the present application can implement all the method steps implemented by the method embodiment in which the execution subject is the electronic device, and can achieve the same technical effect. The parts and beneficial effects in this embodiment that are the same as the method embodiment will not be described in detail here.

[0093] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the general 3D bounding box generation method based on geometric structure provided by the above methods, the method comprising: Generate initial 3D bounding box data based on the pre-trained instance segmentation detector and 2D bounding box data; The initial 3D bounding box data is input into a pre-trained 3D bounding box generation model, and target 3D bounding box data generated by the 3D bounding box generation model is obtained, wherein the 3D bounding box generation model is trained based on a boundary projection loss function, an aspect ratio semantic loss function, and a point cloud and bounding box alignment loss function, and the accuracy of the target 3D bounding box data is greater than that of the initial 3D bounding box data.

[0094] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the general 3D bounding box generation method based on geometric structure provided by the above methods, the method comprising: Generate initial 3D bounding box data based on the pre-trained instance segmentation detector and 2D bounding box data; The initial 3D bounding box data is input into a pre-trained 3D bounding box generation model, and target 3D bounding box data generated by the 3D bounding box generation model is obtained, wherein the 3D bounding box generation model is trained based on a boundary projection loss function, an aspect ratio semantic loss function, and a point cloud and bounding box alignment loss function, and the accuracy of the target 3D bounding box data is greater than that of the initial 3D bounding box data.

[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0097] It should also be noted that the terms "first", "second", etc. in the embodiments of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0098] In the embodiments of the present application, the term "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0099] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0100] In this application, "determine B based on A" means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "determine B based on A and C", "determine B based on A, C and E", "determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "when A meets the first condition, use the first method to determine B"; for example, "when A meets the second condition, determine B", etc.; for example, "when A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "when A meets the first condition, use the first method to determine C, and further determine B based on C", etc.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A general 3D bounding box generation method based on geometric structure, characterized in that: include: Generate initial 3D bounding box data based on the pre-trained instance segmentation detector and 2D bounding box data; The initial 3D bounding box data is input into a pre-trained 3D bounding box generation model, and target 3D bounding box data generated by the 3D bounding box generation model is obtained, wherein the 3D bounding box generation model is trained based on a boundary projection loss function, an aspect ratio semantic loss function, and a point cloud and bounding box alignment loss function, and the accuracy of the target 3D bounding box data is greater than that of the initial 3D bounding box data.

2. The general 3D bounding box generation method based on geometric structure according to claim 1, characterized in that: The training steps of the 3D bounding box generation model include: Determining a confidence score based on the initial 3D bounding box data, the confidence score being used to represent the accuracy of the initial 3D bounding box data; The 3D bounding box generation model is trained based on the boundary projection loss function, the aspect ratio semantic loss function, the point cloud and bounding box alignment loss function and the confidence score.

3. The general 3D bounding box generation method based on geometric structure according to claim 2, characterized in that: The training of the 3D bounding box generation model based on the boundary projection loss function, the aspect ratio semantic loss function, the point cloud and bounding box alignment loss function and the confidence score includes: Determining a center point of the target 3D bounding box based on the confidence score; Determining the range of the target 3D bounding box in 3D space based on the boundary projection loss function; Determining an aspect ratio of the target 3D bounding box based on the aspect ratio semantic loss function; The rotation angle and side length of the target 3D bounding box are determined based on the point cloud and bounding box alignment loss function.

4. The general 3D bounding box generation method based on geometric structure according to claim 2, characterized in that: Determining a confidence score based on the initial 3D bounding box data comprises: Determining a heat map and a centrality based on the initial 3D bounding box data; The confidence score is determined based on the heat map and the centrality.

5. The general 3D bounding box generation method based on geometric structure according to claim 2, characterized in that: The method further comprises: Obtain aspect ratio information based on language macromodel.

6. The general 3D bounding box generation method based on geometric structure according to claim 1, characterized in that: The initial 3D bounding box data includes indoor point cloud data and / or outdoor point cloud data.

7. The general 3D bounding box generation method based on geometric structure according to claim 6, characterized in that: In the case where the initial 3D bounding box data includes indoor point cloud data, the backbone network of the 3D bounding box generation model is FCAF3D.

8. The method for generating a general 3D bounding box based on geometric structure according to claim 6, characterized in that: In the case where the initial 3D bounding box data includes outdoor point cloud data, the backbone network of the 3D bounding box generation model is Centerpoint.

9. A general 3D bounding box generation device based on geometric structure, characterized in that: include: A first generation module, for generating initial 3D bounding box data based on a pre-trained instance segmentation detector and 2D bounding box data; The first acquisition module is used to input the initial 3D bounding box data into a pre-trained 3D bounding box generation model to obtain target 3D bounding box data generated by the 3D bounding box generation model, wherein the 3D bounding box generation model is trained based on a boundary projection loss function, an aspect ratio semantic loss function, and a point cloud and bounding box alignment loss function, and the accuracy of the target 3D bounding box data is greater than that of the initial 3D bounding box data.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the general 3D bounding box generation method based on geometric structure according to any one of claims 1 to 8 is implemented.

Images