Multi-scale light field three-dimensional target detection method and detection device

Abstract

The invention discloses a multi-scale light field three-dimensional target detection method and a detection device, which adopt a branch network to process radiation and density information sampled by a neural radiation field so as to solve the problem of insufficient multi-scale target sensing ability caused by insufficient understanding of scene information in the neural radiation field in a traditional method. Due to different tendentiousness of radiation and density to different scale characteristics, target information of different scales can be effectively mined; meanwhile, according to the target detection method, through layered training and dynamic weighted fusion, the performance of a branch network is enhanced, and the overall multi-scale target detection capability is also improved.

Description

Technical Field

[0001] The present application relates to the field of computer image processing, and in particular to a multi-scale light field three-dimensional target detection method and detection device. Background Art

[0002] Traditional visible light 3D target detection solutions usually rely on point cloud or voxel information. On the one hand, obtaining target point cloud information usually requires professional equipment such as laser radar (LiDAR) or RGB-D cameras. The price and maintenance costs of these devices are high, and the operation process is relatively complicated, which is not conducive to practical promotion. On the other hand, when the model precision is high, the storage cost of point clouds and voxels will show a cubic growth, and the actual model precision is therefore limited, and the effect of target detection will also be affected. In the field of 3D reconstruction, the introduction of neural radiance fields (NeRF) has brought new ideas to solve these problems. NeRF can reconstruct multi-view images taken by ordinary cameras, and its implicit expression and layered sampling characteristics also greatly reduce the number of samples required.

[0003] Inspired by this, some researchers introduced the NeRF model in 3D target detection. There are two main approaches. One is to uniformly sample the pre-trained NeRF within the 3D bounding box of the scene, construct a 3D point cloud containing density and radiation, and then feed it into a single target detection network. The other is to transform NeRF into an end-to-end target detection model and directly predict the position and category of the object by replacing the output with YOLO-style target parameters (such as 3D bounding boxes and category probabilities). Both solutions have achieved certain results, but there are still some shortcomings. The former ignores the volume rendering-based characteristics of NeRF, and the rich high-dimensional information obtained by sampling may cause aliasing of feature extraction; the latter lacks universality for different NeRF models. At the same time, the simultaneous training of the NeRF model and target detection may result in relatively large computing power and time consumption, and the generalization ability will also be limited by the selection of the NeRF model.

[0004] In summary, traditional methods are constrained by the storage and computational issues of high-precision point clouds or voxels, while the current visible light 3D target detection method combined with NeRF has certain deficiencies in terms of effective feature utilization and versatility. Summary of the Invention

[0005] The technical problem solved by this application is: how to improve the detection capability of NeRF-based visible light three-dimensional target detection for objects of different scales and the versatility of input data. To solve this problem, this application provides a multi-scale light field three-dimensional target detection method and detection device.

[0006] According to the first aspect, the present application provides a multi-scale light field three-dimensional target detection method, including: obtaining the three-dimensional coordinate points of the scene through uniform sampling, and generating a high-dimensional feature point cloud including radiance (RGB) and volume density (σ) according to a fully trained neural radiation field model; separating the high-dimensional feature point cloud into radiance feature point cloud and volume density feature point cloud, and performing corresponding preprocessing to adapt to the input requirements of small-scale and large-scale target detection networks; based on the radiance feature point cloud, extracting the fine features of small-scale targets by introducing a Transformer encoder with a local attention mechanism, and outputting a first detection result including the center position, size and category; based on the volume density feature point cloud, filtering out noise points through a density-weighted sampling strategy, and extracting the geometric features of large-scale targets in combination with the PointNet network, and outputting a second detection result including the center position, size and category; according to the confidence of the first detection result and the second detection result, weighted fusion is performed through the Bayesian probability formula to generate the final multi-scale target detection result.

[0007] Furthermore, the three-dimensional coordinate points of the scene are obtained by uniform sampling, and a high-dimensional feature point cloud containing radiation brightness and volume density is generated according to the fully trained neural radiation field model, including: uniform sampling at a preset resolution within the three-dimensional bounding box of the scene to obtain the coordinates (x, y, z) of each sampling point; inputting the coordinates of each sampling point into the fully trained neural radiation field model respectively to obtain the corresponding radiation brightness (RGB) and volume density (σ); and generating a high-dimensional feature point cloud based on the radiation brightness and volume density corresponding to each sampling point.

[0008] Furthermore, the high-dimensional feature point cloud is separated into a radiance feature point cloud and a volume density feature point cloud, and corresponding preprocessing is performed on each to adapt to the input requirements of the small-scale and large-scale object detection networks. This includes: data separation of the high-dimensional feature point cloud to obtain a radiance feature point cloud and a volume density feature point cloud, normalizing and position encoding the former, and adjusting the dimension to adapt to the input structure of the Transformer encoder; and normalizing and adjusting the dimension of the latter to adapt to the input structure of the PointNet network. The Transformer encoder and the PointNet network are respectively configured as object detection networks for extracting small-scale features and large-scale features.

[0009] Furthermore, in the radiation brightness feature point cloud, the Transformer encoder with a local attention mechanism is introduced to extract fine features of small-scale targets, and a first detection result including center position, size and category is output, including: for the point cloud based on radiation brightness features, the radiation brightness feature point cloud is divided into overlapping 3D windows, and the Transformer encoder is used to perform multi-head self-attention calculation in the 3D window to capture local and global contextual association information, and the contextual association information is learned through the target query vector to generate potential target positions, and the classification branch is used to predict the probability of the category, and the regression branch is used to output the center position (x, y, z), size (L, H, W) and category of the potential target position; based on the predicted category probability, the output center position and size obtain the first detection result.

[0010] Furthermore, based on the volume density feature point cloud, noise points are filtered out through a density-weighted sampling strategy, and the geometric features of the large-scale target are extracted in combination with the PointNet network, and a second detection result including the center position, size and category is output, including: performing density-weighted sampling on the volume density feature point cloud, filtering out noise points with a density lower than a preset threshold, retaining high-density areas representing the structure of the large-scale target, and forming a retained point cloud; inputting the retained point cloud into the PointNet network, extracting local features through a multi-layer perceptron, aggregating global shape information through a maximum pooling layer, and finally outputting the center position, size and category of the large-scale target; and obtaining the second detection result based on the output center position, size and category.

[0011] Furthermore, based on the confidence of the first detection result and the second detection result, weighted fusion is performed through the Bayesian probability formula to generate a final multi-scale target detection result, including: calculating the intersection-over-union ratio value of the target box in the first detection result and the second detection result, screening out overlapping target pairs with an intersection-over-union ratio value greater than a preset value, and for the overlapping target pairs, performing weighted fusion processing based on the Bayesian probability formula, retaining overlapping targets with high confidence by weighted fusion of target positions and sizes; for non-overlapping targets, retaining results with confidence higher than a specific threshold, and filtering out low-confidence noise; obtaining the multi-scale target detection result based on the retained overlapping targets and non-overlapping targets.

[0012] Furthermore, the multi-scale light field three-dimensional target detection method also includes: comparing the multi-scale target detection result with the manually labeled true value, calculating the Euclidean distance error of the target center position, the absolute error or relative error of the target frame length, width and height, the target category cross entropy loss error, the target frame intersection-over-union error, and the recall rate and precision of the detected target to form error information; constructing a loss function based on the error information, adjusting the weights of the Transformer encoder and the PointNet network in a backpropagation manner until the loss function converges, and improving the detection accuracy of multi-scale targets by optimizing the model.

[0013] According to the second aspect, the present application provides a detection device, including: a spatial sampling module, which is used to obtain three-dimensional coordinate points of a scene by uniform sampling, and generate a high-dimensional feature point cloud containing radiation brightness and volume density according to a fully trained neural radiation field model; a point cloud separation module, which is connected to the spatial sampling module, and is used to separate the high-dimensional feature point cloud into radiation brightness feature point cloud and volume density feature point cloud, and perform corresponding preprocessing to adapt to the input requirements of small-scale and large-scale target detection networks; a first detection module, which is connected to the point cloud separation module, and is used to extract small-scale targets based on the radiation brightness feature point cloud by introducing a Transformer encoder with a local attention mechanism. The method comprises the following steps: a first detection module, a second detection module, and a second detection module. The first detection module is connected to the point cloud separation module, and is used to filter out noise points based on the volume density feature point cloud through a density weighted sampling strategy, and extract the geometric features of large-scale targets in combination with the PointNet network, and output a second detection result including the center position (x, y, z), size (L, H, W) and category; an information fusion module is connected to the first detection module and the second detection module, and generates a final multi-scale target detection result by weighted fusion through the Bayesian probability formula according to the confidence of the first detection result and the second detection result.

[0014] Furthermore, the detection device also includes an error calculation module and a model optimization module: the error calculation module is connected to the information fusion module signal, and is used to compare the multi-scale target detection result with the manually labeled true value, calculate the target center position Euclidean distance error, the target frame length, width and height absolute error or relative error, the target category cross entropy loss error and the target frame intersection error, and form error information; the model optimization module is used to construct a loss function according to the error information, adjust the weights of the Transformer encoder and the PointNet network in a backpropagation manner until the loss function converges, and improve the detection accuracy of the multi-scale target by optimizing the model; the target center position Euclidean distance error is expressed as The absolute or relative error of the length, width and height of the target frame is expressed as The target category cross entropy loss error is expressed as The target box cross ratio error is expressed as And define the target recall as and the accuracy is Then the loss function of the error information is constructed as

[0015]

[0016] According to a third aspect, the present application provides a computer storage medium storing a program, wherein the program can be processed by a processor to implement the target detection method described in the first aspect.

[0017] The beneficial effects of this application are:

[0018] According to the above-mentioned multi-scale light field three-dimensional target detection method and detection device, since the NeRF model is selected as the carrier of three-dimensional scene information, compared with the method of using point cloud or voxel, only a mobile phone or an ordinary camera is needed to obtain the spatial feature data of different scenes, which reduces the threshold for producing target detection data sets and the storage requirements. Compared with the solution of directly modifying the NeRF model to output target detection results, it can adapt to various rapidly developing neural radiation field models; at the same time, the radiation and density information predicted by the NeRF model are used in different target detection networks during the target detection process, which can fully mine the spatial information stored in NeRF. Among them, the radiation information records the local reflective properties of the object surface (such as color, lighting, texture, etc.), and can capture the fine edges and details of small-scale targets, while large-scale targets (such as buildings, vehicles, large machinery, etc.) usually rely on the overall shape and contour features. The geometric shape of the target is usually more stable than the color and texture, and the density information naturally contains these geometric information, which can reduce the dependence on color noise in the radiation information. Through the targeted design of lightweight branch network structure, we can fully utilize the advantages of both types of information, avoid the optimization conflicts that may be caused by a single network processing small-scale and large-scale targets at the same time, improve the adaptability of the target detection system to targets of different scales, and thus improve the overall detection performance.

[0019] The technical solution of this application introduces the neural radiation light field as the storage medium for the three-dimensional information of the scene, extracts small-scale target information from the sampled radiation information through the self-attention mechanism, and extracts large-scale target information from the sampled density information through the PointNet network, and then dynamically weights and fuses the target information features generated by the two types of feature information to obtain more accurate target detection results. It can be understood that the technical solution of this application not only fully utilizes the advantage of the neural radiation light field in efficiently expressing scene features, but also takes into account both details and overall structure, improving the accuracy and robustness of target detection. At the same time, it reduces the computational complexity through a targeted lightweight network structure, making it suitable for application scenarios with limited resources. Therefore, it achieves a balance between high-efficiency performance and practical application while meeting the needs of multi-scale target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a multi-scale light field three-dimensional target detection method according to an embodiment of the present application;

[0021] Figure 2 This is a flowchart of a multi-scale light field three-dimensional object detection method in another embodiment of the present application;

[0022] Figure 3 A schematic diagram of a network architecture of a preset neural network in one embodiment of the present application;

[0023] Figure 4 This is a flowchart of pre-training of a preset neural network in one embodiment of the present application;

[0024] Figure 5 This is a flowchart of the joint training of a preset neural network in one embodiment of the present application;

[0025] Figure 6 This is a schematic structural diagram of a detection device in one embodiment of the present application;

[0026] Figure 7 This is a schematic structural diagram of a detection device in another embodiment of the present application. DETAILED DESCRIPTION

[0027] In the description of the present invention, it should be understood that the described embodiments are only part of the embodiments of the present invention, not all the embodiments.

[0028] The present application will be further described in detail below through specific implementation methods in conjunction with the accompanying drawings.

[0029] In one embodiment of the present application, a multi-scale light field three-dimensional object detection method is provided, comprising the following steps:

[0030] (1) The three-dimensional coordinate points of the scene are obtained by uniform sampling, and a high-dimensional feature point cloud containing radiation brightness and volume density is generated based on the fully trained neural radiation field model.

[0031] (2) The high-dimensional feature point cloud is separated into the radiation brightness feature point cloud and the volume density feature point cloud, and corresponding preprocessing is performed respectively to adapt to the input requirements of the small-scale and large-scale target detection networks.

[0032] (3) Based on the radiation brightness feature point cloud, the Transformer encoder with local attention mechanism is used to extract the fine features of small-scale targets and output the first detection result including the center position, size and category.

[0033] (4) Based on the volume density feature point cloud, noise points are filtered out through the density weighted sampling strategy, and the geometric features of large-scale targets are extracted in combination with the PointNet network, and the second detection result including the center position, size and category is output.

[0034] (5) Based on the confidence of the first detection result and the second detection result, weighted fusion is performed through the Bayesian probability formula to generate the final multi-scale target detection result.

[0035] For the above method, you can use Figure 1 For details, please refer to the following sections. Figure 1 As shown, it mainly includes steps S11 to S19.

[0036] In step S11, a high-dimensional feature point cloud is obtained by uniformly sampling from the fully trained NeRF model. This can be understood as obtaining the radiation and volume density information of the sampling points through the NeRF model. The NeRF model here can be any neural radiation field model that can generate radiation and density information, and the specific model is not limited.

[0037] In a specific embodiment, uniformly sampling a high-dimensional feature point cloud from a fully trained NeRF model includes the following process:

[0038] (1) Determine the spatial scene range required by the target detection task, emit light along the camera's viewing angle, and split each ray in the 3D scene into multiple sampling points, or obtain sampling points equidistantly along the x-axis, y-axis, and z-axis of the world coordinate system. Ensure that the sampling density covers the spatial distribution of the required scene.

[0039] (2) The sampling coordinates are input into the NeRF model. The model first encodes the coordinates to enhance position perception, and then calculates and predicts the radiance (RGB) information and volume density information through the neural network layer. Each sampling point corresponds to a five-dimensional feature vector (including xyz coordinates, radiation feature vector, and volume density information), ultimately forming a high-dimensional feature point cloud dataset, providing a data foundation for subsequent target detection. In practical applications, the sampling parameters and model settings can be optimized as needed.

[0040] It can be understood that when obtaining the three-dimensional coordinate points of the scene through uniform sampling and generating a high-dimensional feature point cloud containing radiation brightness and volume density according to the fully trained neural radiation field model, uniform sampling can be performed at a preset resolution within the three-dimensional bounding box of the scene to obtain the coordinates of each sampling point; the coordinates of each sampling point are respectively input into the fully trained neural radiation field model to obtain the corresponding radiation brightness and volume density; and a high-dimensional feature point cloud is generated based on the radiation brightness and volume density corresponding to each sampling point.

[0041] In steps S12 and S13, a low-dimensional feature point cloud of the scene's radiance and density information is obtained through preprocessing. The low-dimensional feature point cloud here refers to a simplified representation obtained by dimensionality reduction or feature extraction of the radiance (such as RGB values) and density information sampled from the NeRF model. It retains key features related to the task (such as texture characteristics of small-scale targets and geometric structures of large-scale targets) and removes redundant information, thereby significantly reducing the data dimension while improving computational efficiency and generalization capabilities. The low-dimensional feature point cloud is compact and efficient, and can be used as input to lightweight detection networks (such as Transformer and PointNet), providing more accurate feature support for target detection tasks.

[0042] In a specific embodiment, preprocessing to obtain a low-dimensional feature point cloud of scene information about radiance and density includes the following steps:

[0043] (1) The radiation information is normalized to ensure a consistent data range and improve the stability of the model. Subsequently, the feature dimension is adjusted through linear mapping to match the input requirements of the self-attention network. Finally, position encoding is introduced based on the position information in three-dimensional space to enhance the spatial representation of the radiation information, thereby more efficiently capturing the detailed characteristics of small-scale targets.

[0044] (2) Density information is normalized to eliminate dimensional differences and improve the numerical stability of the data. Next, the dimension of the density feature is adjusted through linear mapping to make it compatible with the input of the PointNet network, thereby effectively extracting the geometric characteristics and global structure of large-scale objects in the scene.

[0045] It can be understood that when the high-dimensional feature point cloud is separated into the radiation brightness feature point cloud and the volume density feature point cloud, and corresponding preprocessing is performed respectively to adapt to the input requirements of the small-scale and large-scale target detection networks, the high-dimensional feature point cloud can be data separated to obtain the radiation brightness feature point cloud and the volume density feature point cloud; the radiation brightness feature point cloud is normalized and adapted to the input structure of the Transformer encoder through position encoding; the volume density feature point cloud is normalized and reorganized through the point cloud format to adapt to the input structure of the PointNet network; the Transformer encoder and PointNet network mentioned here are respectively configured as target detection networks for extracting small-scale features and large-scale features.

[0046] Transformer encoder Figure 3 As shown in the figure 34, it is obtained by stacking multiple layers of multi-head self-attention modules and feedforward neural networks. The layers are optimized through residual connections and layer normalization. Among them, the multi-head self-attention module is used to calculate the similarity between features, capture local and global context associations, and enhance the ability to model small-scale target detail features; the feedforward neural network further processes the features of each sampling point independently to extract high-level feature representations; residual connections ensure the stability of information flow and avoid the gradient vanishing problem; layer normalization improves the numerical stability of model training, thereby improving overall detection performance.

[0047] For the point cloud based on volume density features, noise points with density lower than a preset threshold are filtered out, and high-density areas representing large-scale target structures are retained to form a retained point cloud. The retained point cloud is input into the PointNet network, local features are extracted through a multi-layer perceptron, and global shape information is aggregated through a maximum pooling layer. Finally, the center position (x, y, z), size (L, H, W) and category of the large-scale target are output.

[0048] PointNet network Figure 3 As shown in the labels 37-38, it consists of an input transformation layer, a multi-layer perceptron, and a maximum pooling layer. Among them, the input transformation layer (usually a small T-Net) performs spatial transformation on the coordinates of the input point cloud to enhance the model's robustness to geometric transformations (such as rotation and translation invariance); the multi-layer perceptron (MLP) layer performs nonlinear transformation on the density features, extracts deep feature representations through multiple fully connected layers and activation functions (such as ReLU), and enhances the model's expressive power, thereby better capturing the potential structural information in the data; the maximum pooling layer performs global maximum pooling on the features processed by the MLP layer to aggregate the most significant features of each local area, which helps to aggregate features from local to global, enabling the model to better understand the spatial distribution of large-scale targets.

[0049] Step S14, for the radiation-based feature point cloud, the target information is calculated according to the radiation characteristics, specifically based on the radiation brightness feature point cloud, the Transformer encoder with the local attention mechanism is introduced to extract the fine features of the small-scale target, and the first detection result including the center position, size and category is output.

[0050] In a specific embodiment, for a radiation-based feature point cloud, calculating target information according to radiation features includes the following process:

[0051] (1) The preprocessed radiation features are divided into overlapping 3D windows and multi-head self-attention calculations are performed within the windows. The similarity between each sampling point is calculated by interacting with the query (Q), key (K), and value (V) matrices, and information is aggregated through weighted summation. Through the multi-head mechanism, the model can simultaneously focus on different aspects of the features in different subspaces, enhancing the ability to model global context.

[0052] (2) The features of each sample point are processed independently through a feedforward network (FFN) to further extract high-level features. The FFN contains two fully connected layers and uses the ReLU activation function. Residual connections and layer normalization (LayerNorm) are added after each sub-layer to maintain the stability of information flow and avoid the gradient vanishing problem.

[0053] (3) After processing by the multi-layer encoder, the final features are mapped to the output space of the target detection task through the fully connected layer (FCL), and the category probability and position coordinates of each sampling point are calculated through the Softmax layer. The output includes the classification probability of the target and the predicted bounding box position, which are used for subsequent target recognition and localization tasks.

[0054] It can be understood that when extracting fine features of small-scale targets based on the radiation brightness feature point cloud by introducing a Transformer encoder with a local attention mechanism and outputting a first detection result including the center position, size and category, the radiation brightness feature point cloud can be divided into overlapping 3D windows, and the Transformer encoder can be used to perform multi-head self-attention calculations within the 3D window to capture local and global contextual association information; the contextual association information is learned through the target query vector to generate potential target positions, and the classification branch is used to predict the probability of the category, and the regression branch is used to output the center position and size of the potential target position; based on the predicted category probability, the output center position and size obtain the first detection result.

[0055] In step S15, the target information is calculated based on the density feature of the density-based feature point cloud. Specifically, based on the volume density feature point cloud, noise points are filtered out through a density weighted sampling strategy, and the geometric features of large-scale targets are extracted in combination with the PointNet network. The second detection result including the center position, size and category is output.

[0056] In a specific embodiment, for a density-based feature point cloud, calculating target information based on density features includes the following process:

[0057] (1) The preprocessed density features are filtered out with a density-weighted sampling strategy to remove low-density noise points, and the high-density area point cloud that represents the geometric structure of large-scale objects is preferentially retained.

[0058] (2) The data is sent to the multi-layer perceptron (MLP) layer for nonlinear transformation. The MLP layer extracts deep feature representations through multiple fully connected layers and activation functions (such as ReLU), enhancing the model's expressive power and thus better capturing the potential structural information in the data.

[0059] (3) The features processed by the MLP layer are then pooled globally in the MaxPooling layer to aggregate the most significant features of each local region. The MaxPooling operation helps aggregate features from local to global, enabling the model to better understand the spatial distribution of large-scale objects.

[0060] (4) The pooled features are mapped to the output space of the target detection task through a fully connected layer (FCL). Finally, a softmax layer is used to calculate the class probability and target location prediction value of each sampling point. The softmax layer converts the output into a class probability distribution, thereby determining the classification and location of large-scale targets.

[0061] It can be understood that based on the volume density feature point cloud, noise points are filtered out through the density weighted sampling strategy, and the geometric features of the large-scale target are extracted in combination with the PointNet network. When the second detection result containing the center position, size and category is output, density weighted sampling can be performed on the volume density feature point cloud to filter out noise points with a density lower than a preset threshold, and retain the high-density area representing the large-scale target structure to form a retained point cloud; the retained point cloud is input into the PointNet network, local features are extracted through the multi-layer perceptron, and the global shape information is aggregated through the maximum pooling layer, and finally the center position, size and category of the large-scale target are output; the second detection result is obtained based on the output center position, size and category.

[0062] Step S16, dynamically weighted fusion of target information features generated from the two types of feature information is performed.

[0063] In a specific embodiment, dynamic weighted fusion of target information features generated from two types of feature information includes the following process:

[0064] (1) For the target information from two detection networks, the IoU is calculated for each pair (excluding the combination of target information from the same detection network), and the intermediate-scale target information pairs whose IoU exceeds the threshold are screened out.

[0065] (2) For the selected target pairs, the deviation is assumed to obey a specific statistical distribution, and the fusion weight is assigned based on the Bayesian formula: the deviation is assumed to obey a specific statistical distribution (such as Gaussian distribution), the prior probability obeys a uniform distribution determined by the training data set, and the posterior probability is calculated to guide the weighted average of the position information and the optimal fusion of the size parameters; for unmatched independent targets, they are directly retained or filtered out based on the branch confidence and the preset threshold.

[0066] It can be understood that when the final multi-scale target detection result is generated by weighted fusion through the Bayesian probability formula based on the confidence of the first detection result and the second detection result, the intersection-and-union ratio value of the target box in the first detection result and the second detection result can be calculated, and the overlapping target pairs with the intersection-and-union ratio value greater than the preset value can be screened out; for the overlapping target pairs, weighted fusion processing is performed based on the Bayesian probability formula, and the overlapping targets with high confidence are retained by weighted fusion of the target position and size; for non-overlapping targets, the results with confidence higher than a specific threshold are retained, and low-confidence noise is filtered out; the multi-scale target detection result is obtained based on the retained overlapping targets and non-overlapping targets.

[0067] It should be noted that, taking the target center position fusion as an example, the center positions of the intermediate-scale coincident target pairs to be fused are X1 and X2 respectively, and the center position after fusion is X. Assuming that the deviations obey Gaussian distribution and are independent of each other, and the prior probability obeys uniform distribution, according to the Bayesian formula:

[0068]

[0069] It can be considered as the expectation of the posterior probability μ fused That is, the center position after fusion is X. Since p((X1, X2)|X) is the product of two independent Gaussian distributions, it can be further simplified as:

[0070]

[0071] The product is still a Gaussian distribution, and its mean can be calculated by the following formula:

[0072]

[0073] Furthermore, after fusing the two 3D object detection networks to obtain a multi-scale object detection result, the method further includes: comparing the error information between the obtained multi-scale object detection result and the manually annotated object information in the actual scene, wherein the error information includes the Euclidean distance error of the target center position. Absolute or relative error of the length, width and height of the target frame Target category cross entropy loss error Target box intersection-over-union error And the recall rate of the detection target and precision A loss function is constructed based on the error information, and the weights of the Transformer encoder and the PointNet network are adjusted in a back-propagation manner until the loss function converges, thereby improving the detection accuracy of multi-scale targets by optimizing the model.

[0074] The above error information is determined by the following loss function:

[0075]

[0076] Step S17: Output the fused target information. It is understood that the 3D target detection results include the target center location, the length, width, and height of the target bounding box, and the target category probability distribution. This data is displayed using 3D visualization software, including visualization of the scene sampling point cloud and the cuboid target detection bounding box. The color of the detection bounding box represents the most likely category label for the object.

[0077] In another embodiment, the three-dimensional object detection method may further include steps S11-S17 mentioned above. Figure 2 Steps S18 and S19 shown in FIG.

[0078] Step S18 is located between step S16 and step S17. That is, after dynamically weighted fusion of the target information features generated based on the two types of feature information, it is necessary to compare the error information between the obtained multi-scale target detection results and the manually labeled target information in the actual scene.

[0079] It can be understood that step S18 can be understood as a process of calculating the error between the generated target information and the real target information. The error information includes the Euclidean distance error of the target center position Absolute or relative error of the length, width and height of the target frame Target category cross entropy loss error Target box IoU error And the recall rate of the detection target and precision

[0080] The above error information is determined by the following loss function:

[0081]

[0082] It can be understood that the method proposed in this application also includes: comparing the multi-scale target detection results with the manually annotated true values, calculating the Euclidean distance error of the target center position, the absolute error or relative error of the target box length, width and height, the target category cross entropy loss error, the target box intersection-over-union error, and the recall rate and precision of the detected target to form error information; constructing a loss function based on the error information, adjusting the weights of the Transformer encoder and the PointNet network by backpropagation until the loss function converges, and improving the detection accuracy of multi-scale targets by optimizing the model.

[0083] Step S19: adjusting the network parameters of the preset neural network according to the error information, so as to optimize the network model corresponding to the preset neural network through negative feedback.

[0084] It can be understood that step S19 can be understood as a process of optimizing and updating the model weights. If the error information exceeds a certain threshold, the weight information in the network model corresponding to the preset neural network is adjusted, and then three-dimensional target detection is performed again starting from step S14 and step S15.

[0085] It should be noted that in Figure 2 During the process of processing, since step S18 is added to calculate the error information, a specific judgment can be made in step S17. For example, when the error information is within a certain range, the target detection information is output and displayed according to the fusion result.

[0086] In the 3D object detection method, it is necessary to calculate the target information based on the radiation and density information. This calculation process uses three parts of the preset neural network. During the training process, it is necessary to first pre-train the radiation-based and density-based target detection networks for small-scale and large-scale targets to enable them to initially have the ability to detect small-scale or large-scale targets. Figure 4 ,The pre-training process of the preset branch neural network will be described below.

[0087] Step S41, the preparation phase, involves appropriate preprocessing of the radiation or density information to adapt it to the input requirements of the corresponding object detection network. Furthermore, the network model's operating environment must be configured, and all training parameters must be set, such as the number of training rounds, expected threshold, noise, and training starting point.

[0088] Step S42, check whether the pre-training parameters are read in step S41, specifically whether there is weight data, if so, go to step S43, otherwise go to step S44.

[0089] Step S43: If weight data is available, the matching weight data is loaded into the network model.

[0090] Step S44: In the absence of weight data, randomly initialize weight data and load it into the network model.

[0091] Step S45 , acquiring characteristic information including radiation or density and processing it to obtain target information, for details, please refer to step S14 or step S15 above.

[0092] Step S46, calculate the error between the large-scale or small-scale target results generated by the target detection network and the manually labeled target information in the actual scene. The error information includes the Euclidean distance error of the target center position Absolute or relative error of the length, width and height of the target frame Target category cross entropy loss error Target box IoU error And the recall rate of the detection target and precision For details, please refer to step S18 above.

[0093] In step S47, when the error information is less than the corresponding set value, or the number of training rounds is greater than the corresponding set value, the process proceeds to step S49; otherwise, the process proceeds to step S48.

[0094] In step S48, the network parameters are adjusted according to the error information to optimize the network model corresponding to the preset branch neural network through negative feedback. For details, please refer to step S19 above.

[0095] In step S49, the network model training is considered to be basically qualified, and all parameters of the network model during training can be packaged into a compressed file and stored according to the specified path, mainly saving weight data.

[0096] Step S410, release the occupied memory, clear the temporarily stored training data, and end the training program, that is, the network model training is completed.

[0097] After pre-training, the overall network consisting of two branch networks and the fusion network needs to be jointly trained to enhance the branch network's detection performance for small-scale or large-scale objects and the fusion performance of the fusion network for object pairs with IoU exceeding the threshold. Figure 5 ,The joint training process of the preset overall neural network will be ,explained below.

[0098] Step S51, the preparation stage, requires configuring the operating environment of the network model and performing appropriate preprocessing on the radiation and density information to adapt it to the input requirements of the corresponding target detection network.

[0099] Step S52: Set all training parameters, such as the number of training rounds, expected threshold, noise, training starting point, etc.

[0100] In step S53, the program checks whether the training data exists, is complete, and matches according to the specified path, and then reads the file into memory and converts it into the appropriate data type.

[0101] In step S54, the program generates a model class according to the specified parameters to initialize the network model.

[0102] Step S55, check whether the pre-training parameters are read in step S53, specifically whether there is weight data. If so, go to step S56, otherwise go to step S57.

[0103] Step S56: If weight data is available, the matching weight data is loaded into the network model.

[0104] Step S57: In the absence of weight data, randomly initialize the weight data and load it into the network model.

[0105] Step S58, processing the radiation-based feature information and the density-based feature information in the corresponding branch neural network, and calculating the small-scale and large-scale target information. For details, please refer to the above steps S14 and S15.

[0106] In step S59 , dynamic weighted fusion is performed on the target information features generated based on the two types of feature information. For details, please refer to step S16 above.

[0107] Step S510: compare the error between the multi-scale target detection result and the manually labeled target information in the actual scene. The error information includes the Euclidean distance error of the target center position. Absolute or relative error of the length, width and height of the target frame Target category cross entropy loss error Target box IoU error And the recall rate of the detection target and precision For details, please refer to step S18 above.

[0108] In step S511 , when the error information is smaller than the corresponding set value, or the number of training rounds is larger than the corresponding set value, the process proceeds to step S513 , otherwise, the process proceeds to step S512 .

[0109] Step S512: Adjust the network parameters according to the error information to optimize the preset branch neural network and the network model corresponding to the fusion part through negative feedback. For details, please refer to step S19 above.

[0110] In step S513, the network model training is considered to be basically qualified, and all parameters of the network model during training can be packaged into a compressed file and stored according to the specified path, mainly saving weight data.

[0111] Step S514: release the occupied memory, clear the temporarily stored training data, and end the training program, that is, the network model training is completed.

[0112] According to the visible light multi-scale light field three-dimensional target detection method of the above embodiment, since the NeRF model is selected as the carrier of three-dimensional scene information, compared with the method using point cloud or voxel, only a mobile phone or ordinary camera is needed to obtain spatial feature data of different scenes, thereby reducing the difficulty of producing target detection data sets and storage pressure. Compared with the strategy of directly modifying the NeRF model to output target detection results, this method is more adaptable to the rapid evolution of neural radiation field models; at the same time, during the target detection process, the radiation and density information predicted by the NeRF model can be used to effectively mine the spatial information stored therein. Specifically, the radiation information records the local reflective characteristics of the object surface (such as color, lighting, texture, etc.), which can capture the fine edges and details of small-scale targets; while large-scale targets (such as buildings, vehicles, large machinery, etc.) rely more on overall shape and contour features, and the geometric shape of the target is usually more stable than color and texture. Density information naturally contains these geometric features, thereby reducing dependence on color noise in the radiation information. A specially designed lightweight branch network structure can fully leverage the advantages of both types of information, avoiding the optimization conflicts that can arise when a single network processes both small-scale and large-scale objects. This enhances the object detection system's adaptability to objects of varying scales, thereby improving overall detection performance. This is analyzed in detail below.

[0113] Uniformly sampled from a fully trained NeRF model of reasonable quality, a feature point cloud consisting of radiometric features R and density features D is obtained. After appropriate preprocessing, these feature points are fed into two specialized object detection networks. In the radiometric detection network, the features are first partitioned into overlapping 3D windows. Within these windows, a multi-layer multi-head self-attention mechanism is used to construct similarity relationships consisting of query (Q), key (K), and value (V) matrices. Attention weights are then calculated, and the features are weightedly aggregated. This multi-head mechanism enables the model to capture multidimensional features in different subspaces while enhancing its ability to model global context. The aggregated features are then processed by a feedforward network (FFN), consisting of two fully connected layers with ReLU activation functions. Residual connections and layer normalization are added after each layer to enhance stability and avoid gradient issues. Finally, after multi-layer encoder processing, the features are mapped into object detection outputs, including object classification probabilities and predicted bounding box locations, through fully connected layers. During density feature detection, the preprocessed density features are first filtered out using a density-weighted sampling strategy to remove low-density noise points, prioritizing the retention of high-density point clouds representing the geometric structure of large-scale objects. These features are then passed to a multi-layer perceptron (MLP), which utilizes multiple fully-connected layers and ReLU activation functions to progressively extract deep information and further capture local feature information in the point cloud. The features processed by the MLP are aggregated using a MaxPooling operation to generate a global representation of the most globally salient features. Subsequently, a fully-connected layer maps the global features to the output space of the object detection task, using a softmax function to calculate the object classification probability distribution and bounding box position predictions. The fusion module optimizes the object information generated by the radiation and density features using a dynamic weighting strategy. First, the intersection over union (IoU) between the object information from the two detection networks is calculated, and object pairs whose IoU exceeds a threshold are selected. For these selected pairs, the deviations are assumed to follow a specific statistical distribution, and dynamic adjustment for objects of different scales is achieved using the Bayesian formula. During the training phase, the model is optimized using the cross-entropy loss for object detection and the bounding box position regression loss, and the network parameters are updated through backpropagation. In the inference stage, the trained model is applied to the detection and classification of target objects in the neural radiation field, the final detection result matrix is output, and the point cloud and target detection box are visualized in combination with 3D rendering technology.

[0114] In one embodiment, based on the above-mentioned multi-scale light field three-dimensional target detection method, a detection device is also disclosed. Figure 6 The detection device mainly includes a spatial sampling module 21, a point cloud separation module 22, a first detection module 23, a second detection module 24, and an information fusion module 25, which are described below respectively.

[0115] The spatial sampling module 21 is used to obtain the three-dimensional coordinate points of the scene through uniform sampling, and generate a high-dimensional feature point cloud containing radiation brightness and volume density based on the fully trained neural radiation field model. The spatial sampling module 21 is used to uniformly sample from a fully trained NeRF model with a reasonable quality model, separate the feature point cloud containing radiation information and the feature point cloud containing density information, and perform appropriate preprocessing on each to adapt to the input requirements of the corresponding target detection network. It can be understood that NeRF is a technology for generating high-quality three-dimensional scene rendering, which represents the radiation field and density field in the scene through a neural network. Specifically, the NeRF model inputs the coordinates of a point and the viewing direction in space, and outputs the color (radiation) and volume density of the point. Therefore, radiation and density information can be obtained by spatial sampling of the scene.

[0116] The point cloud separation module 22 is connected to the spatial sampling module 21 and is used to separate the high-dimensional feature point cloud into the radiation brightness feature point cloud and the volume density feature point cloud, and perform corresponding preprocessing to adapt to the input requirements of the small-scale and large-scale target detection networks. For the radiation information, normalization is performed and the feature dimension is adjusted through linear mapping, and then Figure 3 The Positional Encoding layer (labeled 32) in the

[15] layer introduces positional encoding by combining 3D position information, enhancing the spatial representation of radiometric information and more efficiently capturing the detailed features of small-scale objects. Density information is normalized to eliminate dimensionality differences, and the dimensionality of the density features is adjusted through linear mapping to make them compatible with the input of the PointNet network, effectively extracting the geometric characteristics and global structure of large-scale objects in the scene.

[0117] The first detection module 23 is used to extract fine features of small-scale targets based on the radiation brightness feature point cloud by introducing a Transformer encoder with a local attention mechanism, and output a first detection result including the center position, size and category. The first detection module 23 is connected to the point cloud separation module 22, and is used to generate detection results of small-scale targets based on the preprocessed radiation feature point cloud. The detection results include spatial position, target size and type. Specifically, the first detection module 23 can process the coordinates x, y, z of the small-scale target in three-dimensional space, the length (L), width (W), height (H) of the target detection box, and the probability distribution of the target for different categories based on the preprocessed coordinates (x, y, z) and radiation brightness (rgb). Figure 3After passing the position encoding layer (marked 32), the radiation feature point cloud is passed to the first detection module 23, and then passes through the local window segmentation layer (marked 33) to be divided into overlapping 3D windows. It is then processed by two networks within the window. The front network (marked 34) includes a multi-layer multi-head self-attention layer and a feedforward network. The former interacts through the query (Q), key (K) and value (V) matrices to calculate the similarity between each sampling point and aggregate information through weighted summation. The latter processes the features of each sampling point independently to further extract high-level features. In addition, residual connections and layer normalization are added after each sublayer to maintain the stability of information flow and avoid the gradient disappearance problem. The back network (marked 35) is mapped to the output space of the target detection task through the fully connected layer (FCL) and calculates the category probability of each sampling point through the Softmax layer. Finally, the first detection module 23 generates target information for small-scale targets.

[0118] The second detection module 24 is used to filter out noise points based on the volume density feature point cloud through a density weighted sampling strategy, and extract the geometric features of the large-scale target in combination with the PointNet network, and output a second detection result including the center position, size and category. The second detection module 24 is connected to the point cloud separation module 22, and is used to generate the detection results of the large-scale target based on the preprocessed radiation feature point cloud. The detection results include spatial position, target size and type. Specifically, the first detection module 23 can process the coordinates (x, y, z) and radiation brightness (rgb) of the small-scale target in three-dimensional space, the length (L), width (W), height (H) of the target detection box, and the probability distribution of the target for different categories based on the preprocessed coordinates (x, y, z). Figure 3After the preprocessed density feature point cloud is passed to the density-based detection module 24, it first passes through the density weighting layer (marked 36) to filter out low-density noise points and prioritize retaining the high-density area point cloud that represents the geometric structure of large-scale objects. The coordinates of the input point cloud are then spatially transformed through the input transformation layer (marked 37) (usually a small T-Net) to enhance the model's robustness to geometric transformations (such as rotation and translation invariance). It is then processed by two networks. In the front network (marked 38), the density feature is sent to the multi-layer perceptron (MLP) layer for nonlinear transformation. Through multiple fully connected layers and activation functions (such as ReLU), deep feature representations are extracted to enhance the model's expressive power, thereby better capturing the potential structural information in the data. Subsequently, the features processed by the MLP layer are globally pooled through the MaxPooling layer to aggregate the most significant features of each local area, which helps to aggregate features from local to global, enabling the model to better understand the spatial distribution of large-scale targets. In the back network (labeled 39), the pooled features are mapped to the output space of the object detection task through the fully connected layer (FCL), and finally the softmax layer is used to calculate the class probability of each sampling point. Finally, the second detection module 24 generates target information for large-scale objects.

[0119] The information fusion module 25 is used to generate the final multi-scale target detection result by weighted fusion based on the confidence of the first detection result and the second detection result through the Bayesian probability formula. The target information fusion module 25 is connected to the first detection module 23 and the second detection module 24, and is used to perform dynamic weighted fusion based on the target information features generated by the two types of feature information. The calculation process can be referred to Figure 3 In the fusion network (marked 310), for the target information from the two detection networks, the IoU is calculated for each pair (excluding combinations of target information from the same detection network), and intermediate-scale target information pairs whose IoU exceeds a threshold are screened out. The screened target information pairs are based on the Bayesian formula, assuming that the deviation follows a specific statistical distribution (such as a Gaussian distribution) and the prior probability follows a uniform distribution determined by the training dataset. The posterior probability is calculated to guide the weighted average of the position information and the optimal fusion of the size parameters. For unmatched independent targets, they are directly retained or filtered out based on the branch confidence and the preset threshold, ultimately obtaining multi-scale target information.

[0120] In another embodiment, please refer to Figure 7 The detection device disclosed above includes not only a spatial sampling module 21, a first detection module 23, a second detection module 24, and a target information fusion module 25, but also an error calculation module 26 and a model optimization module 27. Each of these modules will be described below.

[0121] The error calculation module 26 is connected to the target information fusion module 25 and is used to compare the error information between the multi-scale target detection results and the manually marked target information in the actual scene. Specifically, the error information includes the Euclidean distance error of the target center position Absolute or relative error of the length, width and height of the target frame Target category cross entropy loss error Target box IoU error And the recall rate of the detection target and precision

[0122] Model optimization module 27 is connected to error calculation module 26, first detection module 23, second detection module 24, and information fusion module 25, and is used to optimize and calculate target detection information for multi-scale objects based on the error information. Specifically, model optimization module 27 adjusts the network parameters of the preset neural network mentioned in first detection module 23, second detection module 24, and target information fusion module 25 based on the error information, thereby optimizing the network model corresponding to the preset neural network through negative feedback. It can be understood that adjusting the parameters of the network model can optimize the detection effect. For target information with large errors, the relevant parameters of the preset neural network can be adjusted to improve detection accuracy.

[0123] Here, the information fusion module 25 is connected to the error calculation module 26. When the error information calculated by the error calculation module 26 is within a preset range, the fused target information is displayed using 3D visualization software, including the visualization of the scene sampling point cloud and the rectangular target detection box. The color of the detection box represents the most likely category label of the object.

[0124] According to the above content, the detection device specifically includes a spatial sampling module 21, a first detection module 23, a second detection module 24, an information fusion module 25, an error calculation module 26 and a model optimization module 27. The functions of each module can correspond to the corresponding content in the introduction of the reference target detection method, and will not be described in detail here.

[0125] It should be noted that in traditional technical solutions, the high-dimensional feature point cloud generated by NeRF sampling is directly input into the target detection network, and the target information generated often only includes objects of a specific scale. However, since the single target detection network has been deeply optimized for specific types of scenes and targets during the training process, the detection effect for objects of more scales may not be ideal. Therefore, in this application, a branched neural network is adopted to detect targets of different scales. The radiation-based network is specifically optimized for small-scale objects, while the density-based network focuses on the detection of large-scale objects.

[0126] It should be noted that in the traditional technical solution, the high-dimensional feature point cloud generated by NeRF sampling is directly input into the target detection network. The input information often includes five-dimensional data such as xyz coordinates, radiation feature vectors and volume density information. These high-dimensional feature information contain too much detail and complexity. When processing larger-scale objects, they are easily interfered by high-dimensional noise, and when processing small-scale objects, they lack sufficient details, resulting in unsatisfactory detection results. In the neural radiation field, radiation information records the reflective properties of the object surface (such as color, lighting, texture, etc.), which is suitable for detecting targets with smaller surface features, can capture subtle local changes, and is suitable for the identification of small-scale objects; density information is suitable for the detection of large-scale objects because it can better express the internal structure and overall morphology of the object, and can provide relatively stable overall features. Therefore, in this application, the radiation information and density information are separated, so the system can process multi-scale targets more accurately, overcoming the performance bottleneck of traditional methods in multi-scale scenarios.

[0127] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer program. When all or part of the functions in the above embodiments are implemented by computer program, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented. In addition, when all or part of the functions in the above embodiments are implemented by computer program, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and saved in the memory of the local device by downloading or copying, or the system of the local device is updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0128] The above content is a further detailed description of the present application in conjunction with specific implementation methods, and the specific implementation of the present application cannot be considered to be limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the inventive concept of the present application.

Claims

1. A multi-scale light field three-dimensional target detection method, characterized in that: include: The 3D coordinate points of the scene are obtained by uniform sampling, and a high-dimensional feature point cloud containing radiation brightness and volume density is generated based on the fully trained neural radiation field model; Separating the high-dimensional feature point cloud into a radiance feature point cloud and a volume density feature point cloud, and performing corresponding preprocessing on each to adapt to the input requirements of small-scale and large-scale target detection networks; Based on the radiance feature point cloud, a Transformer encoder with a local attention mechanism is used to extract fine features of small-scale targets and output a first detection result including center position, size, and category. Based on the volume density feature point cloud, noise points are filtered out through a density weighted sampling strategy, and the geometric features of large-scale targets are extracted in combination with the PointNet network, and a second detection result including the center position, size and category is output; According to the confidence levels of the first detection result and the second detection result, a final multi-scale target detection result is generated by weighted fusion using a Bayesian probability formula.

2. The method according to claim 1, wherein The three-dimensional coordinate points of the scene are obtained by uniform sampling, and a high-dimensional feature point cloud containing radiation brightness and volume density is generated according to the fully trained neural radiation field model, including: Perform uniform sampling at a preset resolution within the three-dimensional bounding box of the scene and obtain the coordinates of each sampling point; The coordinates of each sampling point are input into the fully trained neural radiation field model to obtain the corresponding radiation brightness and volume density; The high-dimensional feature point cloud is generated based on the radiance and volume density corresponding to each sampling point.

3. The method according to claim 1, wherein The high-dimensional feature point cloud is separated into a radiance feature point cloud and a volume density feature point cloud, and corresponding preprocessing is performed on each of them to adapt to the input requirements of small-scale and large-scale target detection networks, including: Performing data separation on the high-dimensional feature point cloud to obtain the radiance feature point cloud and the volume density feature point cloud; Normalizing the radiance feature point cloud and adapting the input structure of the Transformer encoder through position encoding; Normalizing the volume density feature point cloud and reorganizing it into a point cloud format to adapt it to the input structure of the PointNet network; The Transformer encoder and the PointNet network are respectively configured as target detection networks for extracting small-scale features and large-scale features.

4. The method according to claim 3, wherein Based on the radiance feature point cloud, the Transformer encoder with a local attention mechanism is used to extract fine features of small-scale targets, and output a first detection result including center position, size, and category, including: Dividing the radiance feature point cloud into overlapping 3D windows, and using the Transformer encoder to perform multi-head self-attention calculations within the 3D windows to capture local and global contextual association information; The contextual information is learned through the target query vector to generate a potential target location, and the classification branch is used to predict the probability of the category, and the regression branch is used to output the center position and size of the potential target location; The first detection result is obtained based on the predicted category probability, the output center position and size.

5. The method according to claim 3, wherein Based on the volume density feature point cloud, noise points are filtered out through a density weighted sampling strategy, and the geometric features of large-scale targets are extracted in combination with the PointNet network. The second detection result including the center position, size and category is output, including: Performing density-weighted sampling on the volume density feature point cloud, filtering out noise points with a density lower than a preset threshold, retaining high-density areas representing large-scale target structures, and forming a retained point cloud; Input the retained point cloud into the PointNet network, extract local features through a multi-layer perceptron, aggregate global shape information through a maximum pooling layer, and finally output the center position, size and category of large-scale objects; The second detection result is obtained based on the output center position, size and category.

6. The method according to claim 1, wherein The method of generating a final multi-scale target detection result by weighted fusion of the first detection result and the second detection result using a Bayesian probability formula according to the confidence level of the first detection result and the second detection result includes: Calculating the intersection-over-union (IoU) values of the target frames in the first detection result and the second detection result, and screening out overlapping target pairs whose IoU values are greater than a preset value; For the overlapping target pairs, weighted fusion processing is performed based on the Bayesian probability formula, and the overlapping targets with high confidence are retained by weighted fusion of target positions and sizes; For non-coincident targets, the results with confidence higher than a certain threshold are retained, and low-confidence noise is filtered out; The multi-scale object detection result is obtained based on the retained coincident objects and non-coincident objects.

7. The method according to claim 1, wherein Also includes: Compare the multi-scale target detection results with the manually annotated true values, calculate the Euclidean distance error of the target center position, the absolute error or relative error of the length, width and height of the target frame, the cross entropy loss error of the target category, the intersection-over-union error of the target frame, and the recall rate and precision of the detected target to form error information; A loss function is constructed based on the error information, and the weights of the Transformer encoder and the PointNet network are adjusted in a back-propagation manner until the loss function converges, thereby improving the detection accuracy of multi-scale targets by optimizing the model.

8. A detection device, characterized in that: include: The spatial sampling module is used to obtain the three-dimensional coordinate points of the scene through uniform sampling, and generate a high-dimensional feature point cloud containing radiation brightness and volume density based on the fully trained neural radiation field model; A point cloud separation module is used to separate the high-dimensional feature point cloud into a radiance feature point cloud and a volume density feature point cloud, and perform corresponding preprocessing on each to adapt to the input requirements of the small-scale and large-scale object detection networks; A first detection module is configured to extract fine features of small-scale targets based on the radiance feature point cloud by introducing a Transformer encoder with a local attention mechanism, and output a first detection result including a center position, size, and category; A second detection module is configured to filter out noise points based on the volume density feature point cloud using a density-weighted sampling strategy, extract geometric features of large-scale targets in combination with a PointNet network, and output a second detection result including center position, size, and category; The information fusion module is used to generate a final multi-scale target detection result by weighted fusion of the first detection result and the second detection result through a Bayesian probability formula according to the confidence level of the first detection result and the second detection result.

9. The detection device according to claim 8, characterized in that It also includes error calculation module and model optimization module: The error calculation module is connected to the information fusion module signal, and is used to compare the multi-scale target detection result with the manually marked true value, calculate the Euclidean distance error of the target center position, the absolute error or relative error of the length, width and height of the target box, the cross entropy loss error of the target category and the intersection-over-union ratio error of the target box, and form error information; The model optimization module is used to construct a loss function based on the error information, adjust the weights of the Transformer encoder and the PointNet network in a backpropagation manner until the loss function converges, and improve the detection accuracy of multi-scale objects by optimizing the model; The Euclidean distance error of the target center position is expressed as The absolute or relative error of the length, width and height of the target frame is expressed as The target category cross entropy loss error is expressed as The target box cross ratio error is expressed as And define the target recall as and the accuracy is Then the loss function of the error information is constructed as 10. A computer storage medium, characterized in that A program is stored, and when the program is executed by a processor, the multi-scale light field three-dimensional target detection method according to any one of claims 1 to 7 is implemented.

Images