Three-dimensional target detection method and system based on votenet model

Abstract

A 3D target detection method based on a votenet model includes: constructing a votenet model; constructing a point cloud dataset for training the votenet model according to the target of interest; constructing a seed point displacement loss function based on a two-layer nested 3D rectangular box space partition for training the votenet model; constructing other loss functions for training the votenet model based on the original method of the votenet model, including a foreground-background classification loss function, a center offset loss function, a dimension offset loss function, and an orientation angle offset loss function; training the votenet model based on the constructed point cloud dataset and loss function; acquiring point cloud data of a scene to be detected using an RGB-D camera; and outputting a 3D target detection result of the target of interest through the votenet model based on the point cloud data of the scene to be detected. The present invention can effectively reduce the false positive rate of 3D target detection results without increasing model inference delay.

Description

[Technical Field]

[0001] The present invention relates to the field of indoor three-dimensional target detection, and in particular to a three-dimensional target detection method based on a votenet model. [Background technology]

[0002] Target detection algorithms have a wide range of applications and continue to attract attention in the field of computer vision. With the development of deep learning, research on image target detection has made tremendous progress in recent years. Compared to 2D target detection technology, 3D target detection technology, which combines depth information to provide spatial scene information such as target location, orientation, and size, is rapidly developing in the fields of autonomous driving and robotics. Currently, 3D target detection methods based on the votenet model have a relatively high detection recall rate. However, the seed points on non-target surfaces generated by the votenet model tend to appear in the area between multiple targets after displacement processing. When targets are too close to each other, the neighboring points generated by the votenet model may contain features from different targets. Although the foreground features of the collected features are clear, the positioning accuracy is extremely low, resulting in highly reliable false positives. Existing methods typically perform refinement on detection results to reduce the false positive rate, but the additional refinement inevitably requires additional computational effort, resulting in increased inference time. Suppressing the occurrence of false positive targets without increasing inference time is a major challenge facing 3D target detection technology. Summary of the Invention

[0003] The objective of the present invention is to address the shortcomings of the prior art by providing a 3D target detection method based on a votenet model. Compared with other 3D target detection methods based on a votenet model, this method does not increase the model inference time, has a low false positive rate, and a high detection recall rate.

[0004] The objectives of the present invention are achieved by the following technical solutions: A three-dimensional target detection method based on votenet model, which includes the following steps:

[0005] Step 1: Build the vote model.

[0006] Step 2: Construct a point cloud dataset for training the votenet model, aligned with the target of interest.

[0007] Step 3: Construct a seed point displacement loss function based on two-layer nested 3D rectangular box space partitioning to train the votenet model.

[0008] Step 4: Based on the original method of the votenet model, construct other loss functions for training the votenet model, including a foreground-background classification loss function, a center offset loss function, a dimension offset loss function and a direction angle offset loss function.

[0009] Step 5: Train the votenet model based on the constructed point cloud dataset and loss function.

[0010] Step 6: Use an RGB-D camera to acquire point cloud data of the scene to be detected.

[0011] Step 7: Based on the point cloud data of the scene to be detected, the 3D target detection result of the target of interest is output through the votenet model.

[0012] Furthermore, in step 1, a votenet model is constructed based on the construction method described in the votenet paper.

[0013] Furthermore, in step 2, the point cloud dataset refers to a plurality of point cloud data samples including targets of interest and annotation information of a 3D rectangular bounding box of each target of interest in the samples. The point cloud data may be from various types of sensors.

[0014] Furthermore, in step 3 above, for each training sample, the seed point displacement loss function is realized by the following sub-steps:

[0015] (3.1) The point cloud data of the training sample is input into the votenet model, and the seed point coordinates extracted by the votenet model and the coordinate offsets of each seed point predicted by the votenet model are obtained.

[0016] (3.2) The total number of training samples is n t Let i∈[1,n t For the ]th target, the real 3D rectangular bounding box of the target is Box i =(x i ,y i ,z i ,l i ,w i ,h i, θ i ), where x i ,y i ,z i is the center point coordinate of the bounding box, l i ,w i ,h i are the length, width and height of the bounding box, θ i is the orientation angle of the bounding box, and the real 3D rectangular bounding box of the target, Box i For each box, the center coordinates and direction angle of the bounding box are the same, but the length, width, and height are enlarged by k times. i (e) =(x i ,y i ,z i ,k*l i ,k*wi ,k*h i ,θ i )

[0017] (3.3) The total number of seed points is n s Then, traverse all the seed points, j∈[1,n s ]th seed point P j About seed points P j The coordinates in three-dimensional space are (x j ,y j ,z j ) and the seed point P j Fit the entire target to the real 3D rectangular bounding box Box i ,i∈[1,n t ] and find the seed point P j coordinates (x j ,y j ,z j ) is a real 3D rectangular bounding box Box i If the seed point is on the interior or exterior surface of j The target point T j The coordinate values ​​of the real 3D rectangular bounding box Box i The center point coordinate value (x i ,y i ,z i ) seed point P j coordinates (x j ,y j ,z j ) is located inside or on the outer surface of multiple real 3D rectangular bounding boxes, j The coordinate value of the seed point P is the center point of the real 3D rectangular bounding box. j is equal to the center coordinate of the 3D rectangular bounding box that is closest to the

[0018] (3.4) Let n' be the number of seed points that do not have a corresponding target point. s Let j∈[1,n´] be the seed point with no corresponding target point. s]th seed point P j For the seed point P j The three-dimensional rectangular bounding box after the entire expansion is i (e) ,i∈[1,n t ] and find the seed point P j coordinates (x j ,y j ,z j ) the 3D rectangular bounding box after dilation. i (e) If the seed point is on the interior or exterior surface of V, then one corresponding target point V is generated for the seed point. j The target point V j The coordinate values ​​of the seed point P j coordinate value (x j ,y j ,z j ).

[0019] (3.5) Traverse all seed points and find j∈[1,n s ]th seed point P j , the coordinate offset that the votenet model predicts for the seed point is (Δx j ,Δy j ,Δz j ) and the seed point P j The target point T corresponding to j or target point V j If exists, the seed point P j for one corresponding predicted point Q j Generate the predicted point Q j The coordinate value of is (x j +Δx j ,y j +Δy j ,z j +Δz j )

[0020] (3.6) Displacement loss L of the seed point vote-reg Calculate PT k target point T k There exists a seed point Pk , P.T. k The total number of T , PV m the target point V m There exists a seed point P m , PV m The total number of V , D(A,B) is the distance between point A and point B in three-dimensional space, and α is the balancing factor, the specific calculation formula for the displacement loss of the seed point is as follows: JPEG2025536276000002.jpg41122

[0021] The present invention further includes a three-dimensional target detection system based on a votenet model, including:

[0022] A votenet model building module for building a votenet model,

[0023] a point cloud dataset construction module for constructing a point cloud dataset for training a votenet model for a target of interest;

[0024] A seed point displacement loss function construction module that constructs a seed point displacement loss function based on two-layer nested 3D rectangular box space partitioning for training the votenet model.

[0025] an other loss function construction module for constructing other loss functions for training the votenet model, including a foreground-background classification loss function, a center offset loss function, a dimension offset loss function, and an orientation angle offset loss function, based on the original method of the votenet model;

[0026] A Votenet model building module for training a Votenet model based on the constructed point cloud dataset and loss function;

[0027] a point cloud data acquisition module for the scene to be detected, for acquiring point cloud data of the scene to be detected using an RGB-D camera;

[0028] a result output module for outputting a three-dimensional target detection result of the target of interest through the votenet model based on the point cloud data of the scene to be detected;

[0029] The present invention further includes a computer readable storage medium having stored thereon a program that, when executed by a processor, implements the votenet model based three-dimensional target detection method of the present invention.

[0030] The method of the present invention addresses the 3D spatial localization problem of indoor targets by obtaining the 3D spatial coordinates of a specific target within the camera's field of view. The invention implements a seed point displacement loss calculation method based on two-layer nested 3D rectangular box space division to prevent seed points located in the background region near the target in the votenet model from moving toward the target's center region during the displacement phase, thereby preventing the votenet model from generating candidate boxes with high confidence but low target match during the voting point set phase. Compared with other votenet model-based 3D target detection methods, the present invention can effectively reduce the false positive rate of 3D target detection results without increasing model inference delay.

[0031] The beneficial effect of the present invention is that it reduces the probability that background seed points in the votenet model will be displaced midway between multiple targets, thereby reducing the false positive rate of detection results. Compared with seed point segmentation methods based on single-layer 3D rectangular boxes, this method effectively avoids erroneous monitoring signals caused by incomplete annotation of samples, further improving the recall rate of detection results. The present invention solves the high false positive rate problem of current 3D target detection methods based on the votenet model, without introducing extra computational complexity into the inference stage of the votenet model and increasing the inference time of the model. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 is a flow chart of a three-dimensional target detection method according to an exemplary embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram of a target 3D rectangular bounding box shown in an exemplary embodiment. [Figure 3] FIG. 3 is a schematic diagram of the conversion of a depth map into point cloud data shown in an exemplary embodiment. [Figure 4] FIG. 4 illustrates an implementation of the seed point displacement loss function based on two-layer nested 3D rectangular box space partitioning shown in an exemplary embodiment. [Figure 5] FIG. 5 is a system architecture schematic diagram of the present application shown in an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0033] The present invention will be described in more detail below in connection with specific examples. It should be understood that the specific examples described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0034] FIG. 1 is a schematic diagram of a three-dimensional target detection method for a chair-type target in one embodiment of the present application, which includes the following steps:

[0035] Step S101: A voter model is constructed.

[0036] The above votenet model is a votenet model in a publicly published votenet paper, and preferably is a votenet model constructed directly based on the construction method described in the votenet paper.

[0037] Step S102: Construct a point cloud dataset for training a votenet model according to the chair target.

[0038] The point cloud dataset mentioned above refers to a set of point cloud samples containing chair targets and annotation information of the 3D rectangular bounding box of each chair target in the samples. Preferably, the publicly available SUN RGB-D dataset can be used directly.

[0039] In an exemplary embodiment, the three-dimensional rectangular bounding box must completely enclose the target in three-dimensional space, as shown in FIG.

[0040] Step S103: Construct a seed point displacement loss function based on two-layer nested 3D rectangular box space partitioning for training the votenet model.

[0041] Step S104: Construct other loss functions for training the votenet model.

[0042] The above-mentioned other loss functions used in training the votenet model refer to the foreground-background classification loss function, center offset loss function, dimension offset loss function, and orientation angle offset loss function of the votenet model, and preferably may be the foreground-background classification loss function, center offset loss function, dimension offset loss function, and orientation angle offset loss function that construct the votenet model directly based on the construction method described in the votenet paper.

[0043] Step S105: Train a votenet model based on the constructed point cloud dataset and loss function.

[0044] The above training refers to using a backpropagation algorithm to obtain optimized votenet model parameters, and preferably, the SGD algorithm can be used to realize the update process of the votenet model parameters.

[0045] Step S106: The RGB-D camera is used to acquire point cloud data of the scene to be detected.

[0046] The point cloud data refers to point cloud data generated by projecting pixels of a depth image generated by an RGB-D camera into a three-dimensional space based on the internal parameters of the RGB-D camera. In an exemplary embodiment, as shown in FIG. 3, the depth image generated by the RGB-D camera is projected into a three-dimensional space to form point cloud data.

[0047] Step S107: Based on the point cloud data of the scene to be detected, the three-dimensional target detection result of the chair target is output through the votenet model.

[0048] In one exemplary embodiment, as shown in FIG. 4, for each training sample, the calculation steps of the seed point displacement loss function based on two-layer nested 3D rectangular box space partition are as follows:

[0049] Step S201: Input the point cloud data of the training sample into the votenet model, and obtain the seed point coordinates extracted by the votenet model and the coordinate offsets of each seed point predicted by the votenet model.

[0050] Step S202: The training samples are n in total. t Let i∈[1,n t For the ]th target, the real 3D rectangular bounding box of the target is Box i =(x i ,y i ,z i ,l i ,wi ,h i, θ i ), where x i ,y i ,z i is the center point coordinate of the bounding box, l i ,w i ,h i are the length, width and height of the bounding box, and θ i is the orientation angle of the bounding box, and the real 3D rectangular bounding box of the target, Box i For each box, the center coordinates and direction angle of the bounding box are the same, but the length, width, and height are enlarged by k times. i (e) =(x i ,y i ,z i ,k*l i ,k*w i ,k*h i ,θ i )

[0051] The three-dimensional rectangular box Box enlarged k times as above i (e) In this case, k is generally a real number between [1, 2], and may preferably be k=1.5.

[0052] Step S203: Set the total number of seed points to n s Then, traverse all the seed points, j∈[1,n s ]th seed point P j About seed points P j The coordinates in three-dimensional space are (x j ,y j ,z j ) and the seed point P j Fit the entire target to the real 3D rectangular bounding box Box i ,i∈[1,n t ] and find the seed point P j coordinates (x j ,y j ,z j) is a real 3D rectangular bounding box Box i If the seed point is on the interior or exterior surface of j The target point T j The coordinate values ​​of the real 3D rectangular bounding box Box i The center point coordinate value (x i ,y i ,z i ) seed point P j coordinates (x j ,y j ,z j ) is located inside or on the outer surface of multiple real 3D rectangular bounding boxes, j The coordinate value of the seed point P is the center point of the real 3D rectangular bounding box. j is equal to the center coordinate of the 3D rectangular bounding box that is closest to the

[0053] Step S204: The number of seed points without corresponding target points is set to n' s Let j∈[1,n´] be the seed point with no corresponding target point. s ]th seed point P j For the seed point P j The three-dimensional rectangular bounding box after the entire expansion is i (e) ,i∈[1,n t ] and find the seed point P j coordinates (x j ,y j ,z j ) the 3D rectangular bounding box after dilation. i (e) If the seed point is on the interior or exterior surface of V, then one corresponding target point V is generated for the seed point. j The target point V j The coordinate values ​​of the seed point P j coordinate value (x j ,y j ,z j ).

[0054] Step S205: Traverse all seed points and find j∈[1,n s ]th seed point P j , the coordinate offset that the votenet model predicts for the seed point is (Δx j ,Δy j ,Δz j ) and the seed point P j The target point T corresponding to j or target point V j If exists, the seed point P j for one corresponding predicted point Q j Generate the predicted point Q j The coordinate value of is (x j +Δx j ,y j +Δy j ,z j +Δz j )

[0055] Step S206: Displacement loss L of seed points vote-reg Calculate PT k target point T k There exists a seed point P k , P.T. k The total number of T , PV m the target point V m There exists a seed point P m , PV m The total number of V , D(A,B) is the distance between points A and B in three-dimensional space, and α is the balancing factor. The displacement loss of the seed point, L vote-reg The specific calculation formula is as follows: JPEG2025536276000003.jpg41122

[0056] The balancing factor α is used to balance the displacement losses of different types of seed points, and may preferably be α=0.1.

[0057] The present invention further provides a computer-readable storage medium having stored thereon a computer program for use in implementing the three-dimensional target detection method based on the votenet model provided in FIG. 1 above.

[0058] As shown in Figure 5, the present invention further provides a schematic structural diagram of a three-dimensional target detection system based on a votenet model, corresponding to Figure 1. The three-dimensional target detection system based on a votenet model of the present invention includes:

[0059] A votenet model building module for building a votenet model,

[0060] a point cloud dataset construction module for constructing a point cloud dataset for training a votenet model for a target of interest;

[0061] A seed point displacement loss function construction module that constructs a seed point displacement loss function based on two-layer nested 3D rectangular box space partitioning for training the votenet model.

[0062] an other loss function construction module for constructing other loss functions for training the votenet model, including a foreground-background classification loss function, a center offset loss function, a dimension offset loss function, and an orientation angle offset loss function, based on the original method of the votenet model;

[0063] A Votenet model building module for training a Votenet model based on the constructed point cloud dataset and loss function;

[0064] a point cloud data acquisition module for the scene to be detected, for acquiring point cloud data of the scene to be detected using an RGB-D camera;

[0065] a result output module for outputting a three-dimensional target detection result of the target of interest through the votenet model based on the point cloud data of the scene to be detected;

[0066] As shown in Figure 5, at the hardware level, the 3D target detection system based on the votenet model includes a processor, an internal bus, a network interface, an internal memory, and a non-volatile memory, and may also include other hardware required for operation. The processor reads the corresponding computer program from the non-volatile memory into the internal memory and executes it to implement the data collection method described above in Figure 1. Of course, in addition to software implementation, the present invention does not exclude, for example, a logic device or a combination of software and hardware. In other words, the execution entity of the following process flow is not limited to each logic unit, but may also be hardware or a logic device.

[0067] Improvements in a technology can be clearly divided into hardware improvements (e.g., improvements to circuit structures such as diodes, transistors, and switches) and software improvements (improvements to method flow). However, with technological advances, many current method flow improvements can be considered direct improvements to hardware circuit structures. Almost all designers obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that a method flow improvement cannot be realized using a hardware module. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by the user's programming of the device. Designers can "integrate" an entire digital system onto a single PLD through their own programming, eliminating the need for chip manufacturers to design and manufacture dedicated integrated circuit chips.Furthermore, instead of manually creating integrated circuit chips, these types of programs are now often implemented using "logic compiler" software, similar to software compilers used in program development and writing. The original code before compilation must also be written in a specific programming language known as a hardware description language (HDL). There is not just one HDL; there are many, including ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). The most commonly used HDLs today are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that a hardware circuit that implements a logical method flow can be easily obtained by simply programming the method flow into an integrated circuit using some of the hardware description languages ​​mentioned above with a little logical programming.

[0068] The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software, firmware, etc.) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. A memory controller may also be implemented as part of the memory's control logic. Those skilled in the art will recognize that the controller can be implemented not only purely in computer-readable program code form, but also in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc., by logically programming method steps. Therefore, this type of controller can be considered a type of hardware element, and the devices included therein for implementing various functions can also be considered structures within the hardware element. Alternatively, it is also possible to consider that the device for realizing various functions may be a software module that realizes a method, or may be a structure within a hardware component.

[0069] The systems, devices, modules, or units described in the above embodiments can be specifically implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a mobile phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0070] For convenience of description, the above-described apparatus is described as being divided into various units according to their functions. Of course, when implementing the present invention, the functions of each unit can be realized in the same or multiple pieces of software and / or hardware.

[0071] Those skilled in the art should appreciate that embodiments of the present invention may be provided as a method, a system, or a computer program product. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining both software and hardware aspects. The present invention may also take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0072] The present invention has been described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate an apparatus, such that the instructions, executed by the processor of the computer or other programmable data processing device, implement the functions specified in one or more of the flows in the flowcharts and / or one or more blocks in the block diagrams.

[0073] These computer program instructions may also be stored in a computer-readable memory that can be directed to cause a computer or other programmable data processing apparatus to operate in a particular manner to produce an article of manufacture that includes an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0074] These computer program instructions may be implemented in a computer or other programmable data processing apparatus to cause the computer or other programmable device to perform a series of operational steps to generate a computer-implemented process, where the instructions executing on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0075] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and internal memory.

[0076] The internal memory may include forms of volatile memory, random access memory (RAM), and / or non-volatile memory in a computer-readable medium, such as read-only memory (ROM) or flash memory (flash RAM). The internal memory is one example of a computer-readable medium.

[0077] Computer-readable media, including permanent and non-permanent, removable and non-removable media, can achieve storage of information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other internal memory technology, read-only optical disks (CD-ROMs), digital versatile disks (DVDs) or other optical memory, cassette magnetic tape, magnetic tape, magnetic disk memory or other magnetic storage devices, or any other non-transmission media can be used to store information accessible by a computing device. For purposes of this classification, computer-readable media does not include transitory computer-readable media, such as modulated data signals or carrier waves.

[0078] It should be further explained that the terms "comprise," "include," or any variation thereof, cover a non-exclusive inclusion, meaning that a process, method, product, or device that includes a set of elements not only includes those elements, but also includes other elements not expressly listed or that are inherent in such process, method, product, or device. In the absence of a more restrictive context, the phrase "comprising a..." does not exclude the presence of additional identical elements within a process, method, product, or device that includes the elements listed.

[0079] Those skilled in the art will appreciate that embodiments of the present invention may be provided as a method, a system, or a computer program product. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining both software and hardware aspects. The present invention may also take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0080] The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, assemblies, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0081] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts of each embodiment can be mutually referenced, and the emphasis in each embodiment is on the parts that are different from other embodiments. In particular, the system embodiment is described relatively simply because it is basically similar to the method embodiment, so for the relevant parts, you can refer to the description of part of the method embodiment.

[0082] The above description is merely an example of the present invention and is not intended to limit the present invention. Those skilled in the art can make various changes and modifications to the present invention. All modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A three-dimensional target detection method based on a votenet model, comprising: Step 1: Building a Votenet Model; Step 2: constructing a point cloud dataset for training a votenet model according to the target of interest; Step 3: constructing a seed point displacement loss function based on a two-layer nested 3D rectangular box space partition for training the votenet model; Step 4: Based on the original method of the Votenet model, construct other loss functions for training the Votenet model, including a foreground-background classification loss function, a center offset loss function, a dimension offset loss function and a direction angle offset loss function; Step 5: training a votenet model based on the constructed point cloud dataset and loss function; Step 6: acquiring point cloud data of the scene to be detected using an RGB-D camera; Step 7: outputting a three-dimensional target detection result of the target of interest through the votenet model based on the point cloud data of the scene to be detected; A three-dimensional target detection method based on the votenet model.

2. The three-dimensional target detection method based on the votenet model according to claim 1, characterized in that in step 1, the votenet model is constructed based on the construction method described in the votenet paper.

3. 2. The three-dimensional target detection method based on the votenet model of claim 1, wherein in step 2, the point cloud dataset refers to a plurality of point cloud data samples containing targets of interest and annotation information of a three-dimensional rectangular bounding box of each target of interest in the samples.

4. The three-dimensional target detection method based on the votenet model according to claim 3, wherein the point cloud data can be from multiple types of sensors.

5. In step 3, for each training sample, the seed point displacement loss function is Sub-step (3.1) of inputting the point cloud data of the training sample into the Votenet model and obtaining the seed point coordinates extracted by the Votenet model and the coordinate offsets of each seed point predicted by the Votenet model; The training samples are n in total. ｔ targets, i∈[1,n ｔ ]th target, let Box be the real 3D rectangular bounding box of that target. ｉ = (x ｉ , y ｉ , z ｉ , l ｉ , w ｉ , h ｉ， θ ｉ ) where x ｉ , y ｉ , z ｉ is the center point coordinate of the bounding box, ｉ , w ｉ , h ｉ are the length, width, height, and θ of the bounding box. ｉ is the orientation angle of the bounding box, and the real 3D rectangular bounding box of the target, Box ｉ For each box, the center coordinates and direction angle of the bounding box are the same, but the length, width, and height are enlarged by k times. ｉ （ｅ） = (x ｉ , y ｉ , z ｉ , k*l ｉ , k*w ｉ , k*h ｉ , θ ｉ ) and a sub-step (3.2) of generating Let the total number of seed points be n ｓ Let j∈[1,n ｓ ]th seed point P ｊ Regarding seed points P ｊ The coordinates in three-dimensional space are (x ｊ , y ｊ , z ｊ ) and the seed point P ｊ The actual 3D rectangular bounding box of the entire target is ｉ , i∈[1,n ｔ ] and traverse the seed point P ｊ coordinates (x ｊ , y ｊ , z ｊ ) is a real 3D rectangular bounding box Box ｉ If the seed point is on the inner or outer surface of ｊ and generate the target point T ｊ The coordinate values ​​of the real 3D rectangular bounding box Box ｉ The coordinate value of the center point (x ｉ , y ｉ , z ｉ ) and the seed point P ｊ coordinates (x ｊ , y ｊ , z ｊ ) is located inside or on the outer surface of multiple real 3D rectangular bounding boxes, ｊ The coordinate values ​​of the seed point P ｊ a sub-step (3.3) of calculating the coordinates of the center point of the three-dimensional rectangular bounding box closest to the Let n' be the number of seed points that do not have a corresponding target point. ｓ Let j∈[1,n′] be the seed point with no corresponding target point, and traverse all the seed points. ｓ ]th seed point P ｊ For the seed point P ｊ The three-dimensional rectangular bounding box after the whole expansion is ｉ （ｅ） , i∈[1,n ｔ ] and traverse the seed point P ｊ coordinates (x ｊ , y ｊ , z ｊ ) after expansion, the 3D rectangular bounding box Box ｉ （ｅ） If the seed point is on the inner or outer surface of V, then one corresponding target point V is generated for the seed point. ｊ and generate the target point V ｊ The coordinate values ​​of the seed point P ｊ The coordinate value (x ｊ , y ｊ , z ｊ ) and a sub-step (3.4) equal to Traverse all the seed points and find j∈[1,n ｓ ]th seed point P ｊ , the coordinate offset that the votenet model predicts for the seed point is (Δx ｊ , Δy ｊ , Δz ｊ ) and the seed point P ｊ The target point T ｊ or target point V ｊ If exists, the seed point P ｊ , one corresponding predicted point Q ｊ and generate a predicted point Q ｊ The coordinate value of (x ｊ +Δx ｊ , y ｊ +Δy ｊ , z ｊ +Δz ｊ ) sub-step (3.5), Displacement loss L of seed point ｖｏｔｅ－ｒｅｇ The sub-step of calculating PT ｋ at the target point T ｋ There exists a seed point P ｋ , P.T. ｋ The total number of Ｔ , PV ｍ The target point V ｍ There exists a seed point P ｍ , PV ｍ The total number of Ｖ , D(A, B) is the distance between point A and point B in three-dimensional space, and α is the balancing factor, the specific calculation formula for the displacement loss of the seed point is: and a sub-step (3.6) in which The three-dimensional target detection method based on the votenet model according to claim 1.

6. In step 4, the other loss functions used in training the votenet model refer to the foreground-background classification loss function, center offset loss function, dimension offset loss function and directional angle offset loss function of the votenet model, and preferably may be the foreground-background classification loss function, center offset loss function, dimension offset loss function and directional angle offset loss function that construct the votenet model directly based on the construction method described in the votenet paper.

7. The three-dimensional target detection method based on the votenet model according to claim 1, wherein the training in step 5 refers to using a backpropagation algorithm to obtain optimized votenet model parameters.

8. The three-dimensional target detection method based on the votenet model of claim 1, wherein the point cloud data described in step 6 refers to point cloud data generated by projecting pixels of a depth image generated by an RGB-D camera into three-dimensional space based on internal parameters of the RGB-D camera.

9. a votenet model construction module for constructing a votenet model; a point cloud dataset construction module for constructing a point cloud dataset for training a votenet model for a target of interest; a seed point displacement loss function construction module that constructs a seed point displacement loss function based on a two-layer nested 3D rectangular box space partition for training the votenet model; an other loss function construction module for constructing other loss functions for training the Votenet model, including a foreground-background classification loss function, a center offset loss function, a dimension offset loss function, and an orientation angle offset loss function, based on the original method of the Votenet model; a votenet model construction module for training a votenet model based on the constructed point cloud dataset and a loss function; a point cloud data acquisition module for acquiring point cloud data of the scene to be detected using an RGB-D camera; a result output module for outputting a three-dimensional target detection result of the target of interest through the votenet model based on point cloud data of the scene to be detected; A three-dimensional target detection system based on the votenet model.

10. A computer-readable storage medium having stored thereon a program that, when executed by a processor, implements a three-dimensional target detection method based on a votenet model according to any one of claims 1 to 8.

Images