3D object detection method, system, device and storage medium based on physical reasoning

Abstract

The application discloses a kind of 3D target detection method, system, equipment and storage medium based on physical inference, and the method specifically includes the following steps: reading and preprocessing the point cloud data and layout structure data of indoor scene;Feature extraction and global self-attention coding are carried out on point cloud sequence;Then, the initial 3D object prediction bounding box is obtained by using the encoded features and the randomly initialized candidate box sequence after position coding;The stability mechanism and volume repulsion mechanism based on physical inference are used to supervise the initial 3D object prediction bounding box of detection, until the set maximum iteration number is reached, to obtain an end-to-end target detection model.The model can be applied to the point cloud data processed by standard, and the target detection result is predicted.The application can effectively detect the object in three-dimensional indoor scene, and the position and size prediction of the object has high accuracy.

Description

Technical Field

[0001] This invention relates to the field of relation prediction in computer vision, and specifically to a 3D target detection method, system, device, and storage medium based on physical reasoning. Background Technology

[0002] 3D object detection is a key task in environmental perception and scene understanding. It not only provides a research foundation for downstream tasks such as scene graph generation, video or image description, and video question answering, but is also a key technology in applications such as intelligent robot decision-making, autonomous driving, and augmented reality.

[0003] Point cloud data can provide accurate spatial location information, making it ideal for 3D object detection. However, due to its sparsity and objective factors during the acquisition process, it inevitably suffers from problems such as blurred texture information and incomplete shape scanning. This leads to some prediction results that do not conform to physical cognition, such as chairs floating in the air or cabinets passing through walls. Summary of the Invention

[0004] The purpose of this invention is to overcome the problem of distorted prediction results in existing 3D target detection models, and to provide a 3D target detection method, system, device and storage medium based on physical reasoning.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] A 3D object detection method based on physical reasoning includes the following steps:

[0007] Read the point cloud data of the indoor scene and preprocess it into a standard point cloud sequence to obtain a standard point cloud sequence; read the real 3D object bounding box annotation data and layout structure annotation data and preprocess them into a standard bounding box to obtain the real object bounding box and layout structure bounding box.

[0008] Input a standard point cloud sequence, initialize the standard point cloud sequence to obtain an initialized basic feature sequence, and perform global self-attention encoding on the initialized basic feature sequence to obtain encoded features;

[0009] After randomly downsampling the standard point cloud sequence, position encoding and nonlinear mapping are performed to obtain an initial candidate box sequence. Cross-attention decoding is then performed on the encoded features and the initial candidate box sequence to obtain the initial 3D object prediction bounding box. Using the real layout structure bounding box as an aid, the difference between the initial 3D object prediction bounding box and the real object bounding box is calculated by comparing the stability mechanism and volume repulsion mechanism based on physical reasoning as the model loss. The model parameters are updated by backpropagating the model loss until the preset maximum number of iterations is reached, thus obtaining a 3D object detection model based on physical reasoning.

[0010] The point cloud sequence data to be predicted is organized into a standard format, and the target detection results are obtained through a 3D target detection model based on physical reasoning.

[0011] Furthermore, the standard bounding box format is represented by an eight-dimensional vector. The real object bounding boxes and layout structure bounding boxes used in the training process are represented by eight-dimensional vectors. The eight-dimensional vectors include the center coordinates of the bounding box, the length, width, and height of the bounding box, the rotation angle of the object, and the semantic category of the object.

[0012] Furthermore, the input standard point cloud sequence, and the initialization of the standard point cloud sequence to obtain the initial basic feature sequence, includes: first, downsampling the input point cloud sequence using the farthest point sampling algorithm; then, using the nearest neighbor algorithm to retrieve the point cloud within a sphere with a fixed radius centered on the sampling point, forming a point cluster of several nearest neighbor points; and finally, using a three-layer multilayer perceptron to extract the features of the point cluster as the features of the central sampling point, thus obtaining the initial basic feature sequence.

[0013] Furthermore, the global self-attention encoding of the initialized basic feature sequence is performed using a transformer encoder, and the cross-attention decoding of the encoded features and the initialized candidate box sequence is performed using a transformer encoder. The transformer encoder includes six masked encoding layers. In each encoding layer, the initialized basic feature sequence is first normalized, then subjected to a masked multi-head self-attention operation, normalized again, and then subjected to fully connected, random deactivation, and ReLU nonlinear activation operations. A PointNet++ feature aggregation module is applied after the first encoding layer.

[0014] Furthermore, the global self-attention encoding of the initialized basic feature sequence is performed using a transformer encoder, and the cross-attention decoding of the encoded features and the initialized candidate box sequence is also performed using a transformer encoder. The transformer decoder includes eight decoding layers, with the initial candidate box sequence as input and the feature sequence as output. Each decoding layer encodes the position of the candidate box sequence and performs a self-attention operation, adding the result to the original candidate box features. Then, this feature is cross-attentioned with the encoded features, and the final candidate box features are output through random deactivation, ReLU activation function, and fully connected layers. Layer normalization is performed before and after the self-attention and cross-attention operations. Finally, several multilayer perceptron networks are used to parse the parameters of the 3D bounding box of the object and its semantic category.

[0015] Furthermore, the stability mechanism based on physical reasoning, for two interacting objects or layout structures, satisfies the condition that the projection of the center of mass of the supported object onto the contact surface, and calculates the overlapping area and the height difference between the upper and lower object surfaces in the XY plane. If the area of ​​the overlapping area is greater than 0 and the height difference or height overlap does not exceed 10% of the total height of the two objects, then the two objects are considered to have a supporting relationship. The stability score C of the supported object is calculated according to the following formula. stability :

[0016]

[0017] Where S over S is the area of ​​the overlapping region in the XY plane. sup The upper surface area of ​​the support; object t1 is a first-level object, representing all room layout structures; object t2 represents objects placed directly on object t1; and object t3 represents objects that may have interactive support relationships with both object t2 and object t1.

[0018] Loss penalty L of stability mechanism stability Calculate using the following formula:

[0019]

[0020] in, This represents the average stability score of all predicted objects in the scene. The average stability score of all real objects in the scene.

[0021] Furthermore, the volume repulsion mechanism calculates the intersecting volume of the bounding boxes of two objects or layout structures using the following formula:

[0022] V=ΔxΔyΔz

[0023] Δx=x1-x2

[0024] Δy=y1-y2

[0025] Δz=z1-z2

[0026] Where V is the overlap volume of the two object bounding boxes, Δx is the overlap length of the two bounding boxes along the X-axis, Δy is the overlap length of the two bounding boxes along the Y-axis, and Δz is the overlap length of the two bounding boxes along the Z-axis.

[0027] The penalty for the volume repulsion mechanism is calculated using the following formula:

[0028]

[0029] in, Let b be the volume intersection-union ratio of any two objects in the scene.i and b j Let i be the bounding box of any two objects i and j in the scene;

[0030] The model loss is represented by a loss function, which is obtained by weighted summation of the object bounding box center coordinates, size, rotation angle, object semantic category, stability loss, and volume repulsion loss.

[0031] A 3D target detection system based on physical reasoning, comprising:

[0032] The data processing module is used to read point cloud data of indoor scenes and preprocess it into a standard point cloud sequence to obtain a standard point cloud sequence; it also reads real 3D object bounding box annotation data and layout structure annotation data and preprocesses them into standard bounding box format to obtain real object bounding boxes and layout structure bounding boxes.

[0033] The feature encoding module is used to input a standard point cloud sequence, initialize the standard point cloud sequence to obtain an initialized basic feature sequence, and perform global self-attention encoding on the initialized basic feature sequence to obtain encoded features.

[0034] The attention decoding module is used to randomly downsample the standard point cloud sequence, perform position encoding and nonlinear mapping to obtain an initial candidate box sequence, and perform cross-attention decoding on the encoded features and the initial candidate box sequence to obtain the initial 3D object prediction bounding box. Using the real layout structure bounding box as an aid, the difference between the initial 3D object prediction bounding box and the real object bounding box is calculated through stability mechanism and volume repulsion mechanism as the model loss. The model parameters are updated through backpropagation of model loss until the preset maximum number of iterations is reached to obtain a 3D object detection model based on physical reasoning.

[0035] The detection result acquisition module is used to organize the point cloud sequence data to be predicted into a standard format and obtain the target detection result through a 3D target detection model based on physical reasoning.

[0036] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned physics-based 3D target detection method.

[0037] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned 3D target detection method based on physical reasoning.

[0038] Compared with the prior art, the present invention has the following beneficial technical effects:

[0039] This invention proposes a 3D object detection method based on physical reasoning. It introduces a stability mechanism and a volume repulsion mechanism based on physical reasoning, and designs a corresponding loss function to constrain the predicted objects. This makes the prediction results more consistent with the physical laws of real-world scenes, promoting research on 3D object detection tasks. This invention can effectively detect objects in 3D indoor scenes, achieving high accuracy in predicting the position and size of objects. Objects predicted using this invention are not only accurately positioned and reasonably sized, but also conform to the physical laws of the real world.

[0040] This invention provides a 3D object detection method based on physical reasoning. It uses an end-to-end object detection model and the scene layout structure to assist object detection, which can promote object detection in 3D indoor scenes and effectively predict the size and position of objects in 3D scenes.

[0041] This invention performs well on mainstream datasets, and the proposed modules are experimentally verified to be effective. Attached Figure Description

[0042] The accompanying drawings are provided to further understand the invention and constitute a part of this invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0043] Figure 1 This is a flowchart illustrating a 3D target detection method based on physical reasoning according to the present invention.

[0044] Figure 2 This is a schematic diagram of the structure of a 3D target detection system based on physical reasoning according to the present invention.

[0045] Figure 3 This is a schematic diagram of an electronic device structure for a 3D target detection method based on physical reasoning according to the present invention.

[0046] Figure 4 This is a flowchart of the 3D target detection method in an embodiment of the present invention.

[0047] Figure 5(a) illustrates the design principle of the physical stable state of the stability mechanism in the embodiment of the present invention.

[0048] Figure 5(b) illustrates the design principle of the support state for the stability mechanism in this embodiment of the invention.

[0049] Figure 6(a) illustrates the design principle of the volume repulsion mechanism in an embodiment of the present invention, showing the mutual occupation of two objects.

[0050] Figure 6(b) illustrates the design principle of the volume occupancy between the object and the layout structure in the volume repulsion mechanism of this invention.

[0051] Figure 7The results are visualized in the embodiments of the present invention. Detailed Implementation

[0052] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0053] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0054] Example 1

[0055] See Figure 1 A 3D target detection method based on physical reasoning includes the following steps:

[0056] Read the point cloud data of the indoor scene and preprocess it into a standard point cloud sequence to obtain a standard point cloud sequence; read the real 3D object bounding box annotation data and layout structure annotation data and preprocess them into a standard bounding box to obtain the real object bounding box and layout structure bounding box.

[0057] Input a standard point cloud sequence, initialize the standard point cloud sequence to obtain an initialized basic feature sequence, and perform global self-attention encoding on the initialized basic feature sequence to obtain encoded features;

[0058] After randomly downsampling the standard point cloud sequence, position encoding and nonlinear mapping are performed to obtain an initial candidate box sequence. Cross-attention decoding is then performed on the encoded features and the initial candidate box sequence to obtain the initial 3D object prediction bounding box. Using the real layout structure bounding box as an aid, the difference between the initial 3D object prediction bounding box and the real object bounding box is calculated by comparing the stability mechanism and volume repulsion mechanism based on physical reasoning as the model loss. The model parameters are updated by backpropagating the model loss until the preset maximum number of iterations is reached, thus obtaining a 3D object detection model based on physical reasoning.

[0059] The point cloud sequence data to be predicted is organized into a standard format, and the target detection results are obtained through a 3D target detection model based on physical reasoning.

[0060] The standard bounding box format is represented by an eight-dimensional vector. The real object bounding boxes and layout structure bounding boxes used in the training process are also represented by eight-dimensional vectors. The eight-dimensional vector includes the center coordinates of the bounding box, the length, width, and height of the bounding box, the rotation angle of the object, and the semantic category of the object.

[0061] The input standard point cloud sequence is initialized to obtain the initial basic feature sequence. This process involves: first, downsampling the input point cloud sequence using the farthest point sampling algorithm; then, using the nearest neighbor algorithm to retrieve the point cloud within a sphere with a fixed radius centered on the sampling point, forming a point cluster composed of several nearest neighbor points; and finally, using a three-layer multilayer perceptron to extract the features of the point cluster as the features of the central sampling point, thus obtaining the initial basic feature sequence.

[0062] The initial basic feature sequence is encoded globally using a transformer encoder. The encoded features and the initial candidate box sequence are decoded using a cross-attention encoder. The transformer encoder consists of six masked encoding layers. In each encoding layer, the initial basic feature sequence is first normalized, then subjected to masked multi-head self-attention operation, normalized again, and then subjected to fully connected, random deactivation, and ReLU nonlinear activation operations. A PointNet++ feature aggregation module is applied after the first encoding layer.

[0063] Global self-attention encoding of the initialized basic feature sequence is performed using a transformer encoder. Cross-attention decoding of the encoded features and the initialized candidate box sequence is also performed using a transformer encoder. The transformer decoder consists of eight decoding layers. The input is the initial candidate box sequence, and the output is the feature sequence. Each decoding layer encodes the position of the candidate box sequence and performs a self-attention operation. The result is added to the original candidate box features. Then, this feature is cross-attentioned with the encoded features. The final candidate box features are output through random deactivation, ReLU activation function, and fully connected layer. Layer normalization is performed before and after the self-attention and cross-attention operations. Finally, several multilayer perceptron networks are used to parse the parameters of the 3D bounding box of the object and its semantic category.

[0064] The stability mechanism based on physical reasoning considers two interacting objects or structural layouts. It assumes that the projection of the supported object's center of mass onto the contact surface, along with the calculated overlapping area in the XY plane and the height difference between the upper and lower object surfaces, indicates a support relationship between the two objects. If the overlapping area is greater than 0 and the height difference or overlap does not exceed 10% of the total height of the two objects, then the two objects are considered to have a support relationship. The supported object's stability score C is calculated using the following formula. stability :

[0065]

[0066] Where S over S is the area of ​​the overlapping region in the XY plane. supThe upper surface area of ​​the support; object t1 is a first-level object, representing all room layout structures; object t2 represents objects placed directly on object t1; and object t3 represents objects that may have interactive support relationships with both object t2 and object t1.

[0067] Loss penalty L of stability mechanism stability Calculate using the following formula:

[0068]

[0069] in, This represents the average stability score of all predicted objects in the scene. The average stability score of all real objects in the scene.

[0070] The volume repulsion mechanism calculates the intersecting volume of two object or layout bounding boxes using the following formula:

[0071] V=ΔxΔyΔz

[0072] Δx=x1-x2

[0073] Δy=y1-y2

[0074] Δz=z1-z2

[0075] Where V is the overlap volume of the two object bounding boxes, Δx is the overlap length of the two bounding boxes along the X-axis, Δy is the overlap length of the two bounding boxes along the Y-axis, and Δz is the overlap length of the two bounding boxes along the Z-axis.

[0076] Penalty L of volume repulsion mechanism excluion Calculate using the following formula:

[0077]

[0078] in, Let b be the volume intersection-union ratio of any two objects in the scene. i and b j Let i be the bounding box of any two objects i and j in the scene;

[0079] The model loss is represented by a loss function, which is obtained by weighted summation of the object bounding box center coordinates, size, rotation angle, object semantic category, stability loss, and volume repulsion loss.

[0080] See Figure 2 A 3D target detection system based on physical reasoning, comprising:

[0081] The data processing module is used to read point cloud data of indoor scenes and preprocess it into a standard point cloud sequence to obtain a standard point cloud sequence; it also reads real 3D object bounding box annotation data and layout structure annotation data and preprocesses them into standard bounding box format to obtain real object bounding boxes and layout structure bounding boxes.

[0082] The feature encoding module is used to input a standard point cloud sequence, initialize the standard point cloud sequence to obtain an initialized basic feature sequence, and perform global self-attention encoding on the initialized basic feature sequence to obtain encoded features.

[0083] The attention decoding module is used to randomly downsample the standard point cloud sequence, perform position encoding and nonlinear mapping to obtain an initial candidate box sequence, and perform cross-attention decoding on the encoded features and the initial candidate box sequence to obtain the initial 3D object prediction bounding box. Using the real layout structure bounding box as an aid, the difference between the initial 3D object prediction bounding box and the real object bounding box is calculated through stability mechanism and volume repulsion mechanism as the model loss. The model parameters are updated through backpropagation of model loss until the preset maximum number of iterations is reached to obtain a 3D object detection model based on physical reasoning.

[0084] The detection result acquisition module is used to organize the point cloud sequence data to be predicted into a standard format and obtain the target detection results through a 3D target detection model based on physical reasoning.

[0085] See Figure 3 An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a 3D target detection method based on physical reasoning.

[0086] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a 3D target detection method based on physical reasoning.

[0087] Example 2

[0088] This invention proposes a 3D target detection method based on physical reasoning. It utilizes the layout structure of indoor scenes to perform end-to-end 3D indoor scene target detection and proposes two physical constraints: a stability mechanism and a volume repulsion mechanism. This makes the prediction results more consistent with the physical laws of real scenes and can promote research on 3D target detection tasks.

[0089] See Figure 4 An embodiment of the present invention provides a 3D target detection method based on physical reasoning, comprising the following steps:

[0090] Step 1: Read and preprocess point cloud data and layout structure data;

[0091] This invention employs two datasets: the ScanNetV2 dataset and the SUN RGB-D dataset. After input, the data needs to be processed into a standard N×3 sequence format. The real object bounding boxes and layout structure bounding boxes used during training are represented by an eight-dimensional vector [x,y,z,l,w,h,θ,cls], where [x,y,z] are the center coordinates of the box, [l,w,h] are the length, width, and height of the box, θ is the rotation angle of the object around the Z-axis, and cls is the semantic category of the object.

[0092] The ScanNetV2 dataset contains 1513 scan scenes covering 18 object categories. Each scene provides point cloud sequence data and 3D object bounding box annotations. The layout structure data for this dataset comes from SceneCAD, and the planar equations of the layout structure are calculated based on the annotations. The layout structure includes walls, ceilings, and floors in indoor scenes. Since the SceneCAD annotations do not completely cover all scenes in the ScanNetV2 dataset, a total of 1128 scan scenes were actually used in this invention, of which 991 were used as the training set and 157 as the test set.

[0093] The SUN RGB-D dataset provides 10,335 RGB images, depth images, and reconstructed point cloud sequences of indoor scenes. 5,285 pairs of data are used as training data, and the remaining 5,050 pairs are used as test data. Object detection is performed on 10 mainstream object categories. The layout structure data for this dataset is calculated from 2D segmentation data.

[0094] Step 2: Based on the standard point cloud sequence, the features are first initialized through the PointNet++ feature aggregation module, and then self-attention encoding is performed using the transformer encoder;

[0095] For the input point cloud sequence data (N×3), the PointNet++ feature aggregation module initializes the scene features into an N0×C dimensional initial feature sequence. This feature aggregation module first uses the Farthest Point Sampling (FPS) algorithm to downsample the input point cloud (initializing the number of sampling points to N0 = 2048, feature dimension C = 256). Then, it uses the Nearest Neighbor (KNN) algorithm to retrieve the point cloud within a sphere centered on the sampling point and with a fixed radius, thus forming a point cluster of the K nearest neighbors. Finally, a three-layer MLP network is used to extract the features of the point cluster as the features of the central sampling point.

[0096] The initialized basic feature sequence of dimension N0×C is input into the transformer encoder. This encoder consists of six masked encoding layers. Each encoding layer first performs layer normalization on the basic feature sequence, then performs multi-head self-attention, followed by another layer normalization, and then processes it through fully connected layers, random deactivation, and ReLU non-linear activation. The encoding layers do not change the feature dimension. Specifically, a PointNet++ feature aggregation module is applied after the first encoding layer, downsampling the number of feature points to 1024.

[0097] Multi-head self-attention generates an N′×N′ dimensional matrix, which is multiplied by a mask matrix of the same dimension to obtain weights, thus generating new N′×C dimensional features. The mask radius r = [0.4, 0.5, 0.6, 0.8, 1.0, 1.2], and the mask matrix M is a 0-1 matrix that conforms to the following formula:

[0098]

[0099] x i and x j Let |x| be the coordinates of the center points of the bounding boxes of any two objects i and j in the scene. i -x j ||2 represents the Euclidean distance between the center coordinates of the two bounding boxes.

[0100] Step 3: Input the encoded features and the initial candidate box sequence after position encoding into the transformer decoder to obtain the initial 3D object prediction bounding boxes;

[0101] Random downsampling is performed on the point cloud sequence N0×3 from step 2, and then the initial candidate box sequence B×d is obtained through position encoding and nonlinear mapping, where B is the number of candidate boxes and d is the feature dimension of the candidate box sequence.

[0102] Similar to the standard 3DETR decoder, the entire decoder consists of eight decoding layers. The input is an initial candidate box sequence B×d, and the output is a feature sequence B×d′. Specifically, each decoding layer first encodes the positions of the candidate box sequence and performs a self-attention operation, adding the result to the original candidate box features. Then, this feature is cross-attentioned with the encoded features. Finally, the final candidate box features are output through random dropout, ReLU activation, and a fully connected layer. Notably, all self-attention and cross-attention operations are normalized before and after; layer normalization is used in this step. Finally, multiple MLP networks are used to parse the parameters of the object's 3D bounding box and its semantic category.

[0103] Step 4: Using a physics-based stability mechanism and a volume repulsion mechanism, and with real layout structure data as an aid, calculate the similarity between the initial 3D object prediction bounding box and the real object bounding box. Update the model parameters through backpropagation until the set maximum number of iterations is reached, and obtain the physics-based 3D object detection model.

[0104] In this embodiment, the number of candidate boxes parsed in step 3 is set to 256. All candidate boxes will be matched with the ground truth boxes using Hungarian matching to select the final B object bounding boxes. The matching cost is obtained by weighting the object category, the bounding box center coordinates, and the intersection volume.

[0105] The learning process for the bounding boxes described above is only constrained by the center coordinates, size, and rotation angle of the boxes. There is no interaction between objects; objects are within the scene, but the scene's layout does not impose any constraints on object prediction. Furthermore, due to the sparsity and irregularity of point clouds, as well as objective factors during data acquisition, they cannot provide dense texture and shape information, and may even have local data gaps. These factors can cause physical conflicts between objects, such as bounding boxes that occupy each other's space or unstable prediction boxes. To strengthen the connections between objects, this invention proposes two physical constraint mechanisms: a stability mechanism and a volume repulsion mechanism.

[0106] The stability mechanism operates between object-object and object-layout structures, aiming to ensure that all objects are in a physically stable state. Referring to Figure 5(a), in this embodiment of the invention, the object satisfies conditions 1) the object is in equilibrium and has support, and 2) the projection point M of the object's center of mass M onto the supporting object... ′ If the object is located exactly at the point of contact or exactly on the contact surface, then the object is stable. A stable state can be expressed by the following formula:

[0107]

[0108] Where δ(X), δ(P), and δ(M) are indicator functions. δ(Z) = 1 when condition Z is satisfied, and δ(Z) = 0 otherwise. S(B,X,P,M) is the steady-state expression.

[0109] According to the definition of stability, the stability mechanism of this invention uses the matched predicted object bounding boxes as input to calculate the stability score between each object-object and object-layout structure. Specifically, for the bounding boxes of two object / layout structures, the overlapping area in the XY plane and the height difference between the upper and lower object surfaces are calculated. If the area of ​​the overlapping area is greater than 0 and the height difference / height overlap does not exceed 10% of the sum of the heights of the two objects, then the two objects are considered to have a support relationship. An object can have one or more supports. If object B1 is supported by multiple objects, as shown in Figure 5(b), where B2 and B3 are simultaneously supports of B1, then a large cuboid B′ capable of enclosing all supports is considered to be present. 23 The object is the sole support. For an object whose center of mass projects onto the contact surface, the stability score C is calculated using the following formula. stability :

[0110]

[0111] Where S over S is the area of ​​the overlapping region in the XY plane. sup This represents the upper surface area of ​​the supporting structure. Object t1 is a primary object, representing all room layout structures. Object t2 represents objects directly placed on object t1, such as beds and cabinets. Object t3 represents objects that may have interactive support relationships with both objects t2 and t1, such as chairs and sinks.

[0112] The volume exclusion mechanism operates between object-object and object-layout structures, aiming to ensure that all objects are volume-exclusive. Two objects are defined as volume-exclusive when their bounding box overlap volume V ≤ 0. Figure 6(a) shows the case where two objects occupy each other's space, and Figure 6(b) shows an example of volume occupancy between an object and a layout structure. The specific volume occupancy can be calculated using the following formula:

[0113]

[0114] Where Δx is the overlap length of the two bounding boxes along the X-axis, Δy is the overlap length of the two bounding boxes along the Y-axis, and Δz is the overlap length of the two bounding boxes along the Z-axis;

[0115] For the entire model learning process, the loss function L phy-3detr The difference between the predicted bounding box and the ground truth bounding box is represented by the object bounding box prediction loss L. obj Stability loss L staand volume repulsion loss L exclu We get λ by weighted summation. o , λ s , λ e These are the weights for the object prediction loss, stability loss, and volume repulsion loss, respectively. The specific calculation method is as follows:

[0116]

[0117] in, c is the center coordinate of the object. c is the predicted value, and c is the actual value. And d is the size of the object, s represents the semantic category of the object, and a c A 12-sector classification of the object's rotation angle, a r The residual of the object's rotation angle is expressed by the formula... This indicates that the object rotation angle residual uses Huber loss. Measurement. λ c , λ d , λ ar , λ ac , λ s These are the weighted weights of the object's center coordinates, bounding box size, rotation angle (12 sector classification), rotation angle residual, and object semantic category. This represents the average stability score of all predicted objects in the scene. The average stability score of all real objects in the scene. Let b be the volume intersection-union ratio of any two objects in the scene. i and b j Let i be the bounding box of any two objects i and j in the scene.

[0118] Step 5: Organize the point cloud sequence data to be predicted into a standard format, and calculate the prediction result using the above target detection model. Figure 7 The detection results are visualized.

[0119] The model proposed in this invention is end-to-end during the training phase, with a total of 1080 training rounds, i.e., the maximum number of iterations. The batch size is 8 for the ScanNet dataset and 16 for the SUN RGB-D dataset. The loss function is a weighted sum of candidate box parameters, candidate box semantic categories, stability loss, and volume exclusion loss. The Adam algorithm is used as the model optimizer for backpropagation, with an initial learning rate of 0.0005 on the ScanNetV2 dataset and 0.001 on the SUN RGB-D dataset. For data augmentation, the input point cloud is randomly flipped horizontally; the point cloud is rotated, with the rotation angle following a range of [-5]. ° 5 ° A uniform distribution on [ ], for the SUN RGB-D dataset, the rotation angle follows [-30 ] . ° 30 ° The point cloud is scaled with a scaling factor that follows a uniform distribution on [0.9, 1.1]. The ablation results of the physical mechanism constraints are shown in Table 1.

[0120] Table 1. Ablation results demonstrating the effectiveness of physical mechanism constraints (all results are recorded as a percentage).

[0121]

[0122] A 3D target detection system based on physical reasoning, comprising:

[0123] The data processing module is used to read point cloud data of indoor scenes and preprocess it into a standard point cloud sequence to obtain a standard point cloud sequence; it also reads real 3D object bounding box annotation data and layout structure annotation data and preprocesses them into standard bounding box format to obtain real object bounding boxes and layout structure bounding boxes.

[0124] The feature encoding module is used to input a standard point cloud sequence, initialize the standard point cloud sequence to obtain an initialized basic feature sequence, and perform global self-attention encoding on the initialized basic feature sequence to obtain encoded features.

[0125] The attention decoding module is used to randomly downsample the standard point cloud sequence, perform position encoding and nonlinear mapping to obtain an initial candidate box sequence, and perform cross-attention decoding on the encoded features and the initial candidate box sequence to obtain the initial 3D object prediction bounding box. Using the real layout structure bounding box as an aid, the difference between the initial 3D object prediction bounding box and the real object bounding box is calculated through stability mechanism and volume repulsion mechanism as the model loss. The model parameters are updated through backpropagation of model loss until the preset maximum number of iterations is reached to obtain a 3D object detection model based on physical reasoning.

[0126] The detection result acquisition module is used to organize the point cloud sequence data to be predicted into a standard format and obtain the target detection result through a 3D target detection model based on physical reasoning.

[0127] In summary, this invention is applicable to point cloud-based indoor scenes, capable of efficiently and accurately detecting the position, size, and rotation angle of various objects. This invention proposes a physics-based stability mechanism and a volume repulsion mechanism, and designs corresponding loss functions to constrain the predicted objects. This invention uses an end-to-end object detection model and leverages the scene's layout structure to assist object detection. This invention performs excellently on mainstream datasets, and experimental verification of the proposed modules demonstrates their effectiveness.

[0128] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0130] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A 3D target detection method based on physical reasoning, characterized in that, Includes the following steps: Read the point cloud data of the indoor scene and preprocess it into a standard point cloud sequence to obtain a standard point cloud sequence; read the real 3D object bounding box annotation data and layout structure annotation data and preprocess them into a standard bounding box to obtain the real object bounding box and layout structure bounding box. Input a standard point cloud sequence, initialize the standard point cloud sequence to obtain an initialized basic feature sequence, and perform global self-attention encoding on the initialized basic feature sequence to obtain encoded features; After randomly downsampling the standard point cloud sequence, position encoding and nonlinear mapping are performed to obtain an initial candidate box sequence. Cross-attention decoding is then performed on the encoded features and the initial candidate box sequence to obtain the initial 3D object prediction bounding box. Using the real layout structure bounding box as an aid, the difference between the initial 3D object prediction bounding box and the real object bounding box is calculated by comparing the stability mechanism and volume repulsion mechanism based on physical reasoning as the model loss. The model parameters are updated by backpropagating the model loss until the preset maximum number of iterations is reached, thus obtaining a 3D object detection model based on physical reasoning. The stability mechanism based on physical reasoning states that for two interacting objects or layout structures, the projection of the center of mass of the supported object onto the contact surface is satisfied, and the overlapping area and the height difference between the upper and lower object surfaces are calculated in the XY plane. If the area of ​​the overlapping area is greater than 0 and the height difference or height overlap does not exceed 10% of the total height of the two objects, then the two objects are considered to have a supporting relationship. The stability score of the supported object is calculated according to the following formula. : in Let be the area of ​​the overlapping region in the XY plane. The upper surface area of ​​the support; The object is a level-one object, representing the layout and structure of all rooms. The object represents being placed directly. Objects on an object, Object representation and Objects and All objects may have interactive support relationships; Loss of stability mechanism The punishment is calculated using the following formula: in, This represents the average stability score of all predicted objects in the scene. The average stability score of all real objects in the scene; The volume repulsion mechanism calculates the intersecting volume for the bounding boxes of two objects or layout structures. The volume repulsion mechanism calculates the intersecting volume of two object or layout bounding boxes using the following formula: Where V is the overlap volume of the bounding boxes of the two objects. Let x be the overlap length of the two bounding boxes along the X-axis. This represents the overlap length of the two bounding boxes along the Y-axis. The overlap length of the two bounding boxes along the Z-axis; The penalty of the volume repulsion mechanism Calculate using the following formula: in, Let be the volume intersection-union ratio of any two objects in the scene. and For any two objects in the scene and The bounding box; The model loss is represented by a loss function, which is obtained by weighted summation of the object bounding box center coordinates, size, rotation angle, object semantic category, stability loss, and volume repulsion loss. The point cloud sequence data to be predicted is organized into a standard format, and the target detection results are obtained through a 3D target detection model based on physical reasoning.

2. The 3D target detection method based on physical reasoning according to claim 1, characterized in that, The standard bounding box format is represented by an eight-dimensional vector. The real object bounding boxes and layout structure bounding boxes used in the training process are represented by eight-dimensional vectors. The eight-dimensional vectors include the center coordinates of the bounding box, the length, width, and height of the bounding box, the rotation angle of the object, and the semantic category of the object.

3. The 3D target detection method based on physical reasoning according to claim 1, characterized in that, The input standard point cloud sequence is initialized to obtain the initial basic feature sequence by: firstly, downsampling the input point cloud sequence using the farthest point sampling algorithm; then, using the nearest neighbor algorithm to retrieve the point cloud within a sphere with a fixed radius centered on the sampling point, forming a point cluster of several nearest neighbor points; and finally, using a three-layer multilayer perceptron to extract the features of the point cluster as the features of the central sampling point, thus obtaining the initial basic feature sequence.

4. The 3D target detection method based on physical reasoning according to claim 1, characterized in that, The global self-attention encoding of the initialized basic feature sequence is completed using a transformer encoder. The cross-attention decoding of the encoded features and the initialized candidate box sequence is completed using a transformer encoder. The transformer encoder includes six masked encoding layers. In each encoding layer, the initialized basic feature sequence is first normalized, then subjected to a masked multi-head self-attention operation, normalized again, and then subjected to fully connected, random deactivation, and ReLU nonlinear activation operations. A PointNet++ feature aggregation module is applied after the first encoding layer.

5. The 3D target detection method based on physical reasoning according to claim 1, characterized in that, The global self-attention encoding of the initialized basic feature sequence is performed using a transformer encoder, and the cross-attention decoding of the encoded features and the initialized candidate box sequence is also performed using a transformer encoder. The transformer decoder includes eight decoding layers, with the initial candidate box sequence as input and the feature sequence as output. Each decoding layer encodes the position of the candidate box sequence and performs a self-attention operation. The result is added to the original candidate box features, and then the added features are cross-attentioned with the encoded features. The final candidate box features are output through random deactivation, ReLU activation function, and fully connected layers. Layer normalization is performed before and after the self-attention and cross-attention operations. Finally, several multilayer perceptron networks are used to parse the parameters of the 3D bounding box of the object and its semantic category.

6. A 3D target detection system based on physical reasoning, using the 3D target detection method based on physical reasoning as described in any one of claims 1-5, characterized in that, include: The data processing module is used to read point cloud data of indoor scenes and preprocess it into a standard point cloud sequence to obtain a standard point cloud sequence; it also reads real 3D object bounding box annotation data and layout structure annotation data and preprocesses them into standard bounding box format to obtain real object bounding boxes and layout structure bounding boxes. The feature encoding module is used to input a standard point cloud sequence, initialize the standard point cloud sequence to obtain an initialized basic feature sequence, and perform global self-attention encoding on the initialized basic feature sequence to obtain encoded features. The attention decoding module is used to randomly downsample the standard point cloud sequence, perform position encoding and nonlinear mapping to obtain an initial candidate box sequence, and perform cross-attention decoding on the encoded features and the initial candidate box sequence to obtain the initial 3D object prediction bounding box. Using the real layout structure bounding box as an aid, the difference between the initial 3D object prediction bounding box and the real object bounding box is calculated through stability mechanism and volume repulsion mechanism as the model loss. The model parameters are updated through backpropagation of model loss until the preset maximum number of iterations is reached to obtain a 3D object detection model based on physical reasoning. The detection result acquisition module is used to organize the point cloud sequence data to be predicted into a standard format and obtain the target detection result through a 3D target detection model based on physical reasoning.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a 3D target detection method based on physical reasoning as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, implements a 3D target detection method based on physical reasoning as described in any one of claims 1-5.

Images