Nerve radiation field-based few-sample target detection method and system

Abstract

The invention relates to the technical field of three-dimensional modeling, in particular to a few-sample target detection method and system based on a neural radiation field, and the method comprises the steps: collecting a to-be-modeled object, and carrying out the processing of the to-be-modeled object, and constructing a target data set; setting multi-resolution Hash mapping through the three-dimensional coordinates, establishing a Hash table, and obtaining mixed features based on the Hash table; performing neural network reasoning by using MLP to obtain density and RGB color, generating light and delimiting a light range, and determining a sampling mode to construct a probability density function; rendering to obtain a multi-view texture image, and fusing and outputting a three-dimensional model by using Poisson reconstruction; constructing two-dimensional images of different angles, generating a training data set, training the three-dimensional model by adopting the training data set, obtaining a to-be-detected sample, and inputting the to-be-detected sample into the trained three-dimensional model to obtain a detection result; through a cooperation mechanism of explicit three-dimensional reconstruction and implicit feature coding, a three-dimensional model is reconstructed to carry out few-sample target detection, and the view angle constraint of traditional two-dimensional detection is broken through.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional modeling, in particular to a few-shot object detection method and system based on neural radiance fields. BACKGROUND

[0002] The target detection technology has been widely used in automatic driving, intelligent security, industrial quality inspection and other fields. However, the implementation of this technology usually relies on a large amount of labeled data, which is difficult to obtain in most real scenarios. For example, rare lesion recognition of medical images, capture of medium and low frequency fault patterns in industrial defect detection, and classification of specific cultural relics in cultural relic protection. These tasks often cannot be automatically detected due to the lack of labeled samples and uneven class distribution. Therefore, when a deep learning model is over-fitted or performance bottlenecked due to lack of data, the importance of few-shot object detection (FSOD) technology becomes more prominent.

[0003] Compared with the existing traditional target detection which relies on big data and strong supervision, the few-shot object detection technology realizes fast and accurate recognition through a small amount of labeled samples. The key is how to extract effective visual knowledge from limited information to build a robust representation of unseen classes. Current mainstream FSOD technology mainly focuses on two directions: one is to alleviate the sample scarcity through data augmentation or synthesis technology; the other is to design a model architecture for cross-task knowledge transfer. However, these technologies still face the problem of model generalization bottleneck caused by data scarcity and the inherent perspective limitation of two-dimensional images. These technologies will face the problem of model generalization bottleneck caused by data scarcity and the inherent perspective limitation of two-dimensional images, which will affect the accuracy of the final target detection. SUMMARY

[0004] In order to solve the technical problem of the existing few-shot object detection technology that the model generalization bottleneck and the inherent perspective limitation of two-dimensional images affect the accuracy of target detection, the purpose of the present application is to provide a few-shot object detection method based on neural radiance fields. The technical solution adopted is as follows:

[0005] Collect the object to be modeled and process it to build a target data set, obtain the camera pose and three-dimensional coordinates of any target image in the target data set, and output the camera parameters;

[0006] Set multi-resolution hash mapping through three-dimensional coordinates, establish a hash table, extract features based on the hash table, and use trilinear interpolation for linear fusion to obtain mixed features;

[0007] The mixed features are inferred by multiple MLPs to obtain density and RGB color, a light ray is generated by combining camera pose and camera parameters, a light ray range is determined based on the light ray, a sampling mode is determined to construct a probability density function;

[0008] A multi-view texture map is obtained by rendering the probability density function and the RGB color, and a three-dimensional model is output by using Poisson reconstruction fusion;

[0009] A two-dimensional image of different angles is constructed to generate a training data set, the three-dimensional model is trained using the training data set, a sample to be detected is obtained, and the detection result is obtained by inputting the sample to the trained three-dimensional model.

[0010] Preferably, the object to be modeled is collected and processed to construct a target data set, and the camera pose and three-dimensional coordinates of any target image in the target data set are obtained, including:

[0011] A video of the object to be modeled is taken, key frames of the video are extracted by FFmpeg, and blurred frames are removed by Laplacian variance filtering to construct the target data set;

[0012] The SIFT module is called to extract local feature points from the target data set, generate feature descriptors, and establish a correspondence between the feature points;

[0013] The essential matrix is obtained, the correspondence between the feature points is filtered, the incremental SFM reconstruction is performed, the camera pose is added in sequence and the feature points are triangulated, the camera pose and three-dimensional coordinates of any target image in the target data set are obtained, and the sparse point cloud and camera parameters are output.

[0014] Preferably, a multi-resolution hash mapping is set by the three-dimensional coordinates, a hash table is established, features are extracted based on the hash table, and linear fusion is performed using trilinear interpolation to obtain mixed features, including:

[0015] The hash index is calculated by the three-dimensional coordinates, a resolution network is established, and the hash table is determined;

[0016] Any pixel point in each layer of the resolution network is defined as a target point, the three-dimensional coordinates are normalized, and multiple voxel vertices surrounding the target point are located by the hash table;

[0017] Based on the hash index, a multidimensional feature vector is obtained from the hash table for each voxel vertex;

[0018] The features of the target point are determined by trilinear interpolation for the hash table, and the mixed features are obtained by splicing.

[0019] Preferably, the hash index is calculated by the three-dimensional coordinates, the resolution network is established, and the hash table is determined, including:

[0020] The hash index is calculated by three-dimensional coordinates, and the corresponding calculation formula is:

[0021]

[0022] wherein, represents the hash index; represents the bitwise XOR; represents the i-th dimension in the three-dimensional coordinates; represents a large prime number, i.e. the prime number of the i-th dimension assigned to the target point; represents the hash table size; A plurality of layers of resolution networks are set, the resolution of each layer is sorted from small to large according to the equal ratio series growth amount, the minimum resolution and the maximum resolution are determined respectively, the hierarchical index of the current layer is calculated, and the hash table is established, and the corresponding calculation formula is:

[0023]

[0024]

[0025] wherein, represents the hierarchical index of the i-th layer; represents the minimum resolution; represents the maximum resolution; represents the total number of layers of the resolution network; represents the i-th layer in the resolution network. Preferably, the hash table is used to determine the features of the target point by trilinear interpolation, and the mixed features are obtained by splicing, including:

[0026] The trilinear interpolation is used, and the corresponding calculation formula is:

[0027]

[0028]

[0029] wherein, represents the interpolated feature of the current normalized target point; represents the interpolated feature of the current normalized target point; represents the position data corresponding to the normalized three-dimensional coordinates, respectively; is the weight of trilinear interpolation; represents the feature of the current normalized target point before interpolation;

[0030] The features of all layers of the resolution network are obtained, and the mixed features are obtained by splicing, and the corresponding calculation formula is:

[0031] ​​​​​​

[0032] wherein, denotes the mixed feature; denotes the interpolated feature of the first layer of the resolution network; denotes the interpolated feature of the first layer of the resolution network; denotes the interpolated feature of the first layer of the resolution network; denotes the total number of layers of the resolution network.

[0033] Preferably, the mixed feature is inferred by a plurality of MLPs to obtain the density and the RGB color, a ray is generated by combining the camera pose with the camera parameter, a ray range is demarcated based on the ray, a sampling manner is determined to construct a probability density function, including:

[0034] The mixed feature is inferred by a plurality of MLPs to obtain the density and the RGB color in sequence;

[0035] Based on any image pixel of the target image, a ray is generated by combining the camera parameter and the camera pose;

[0036] The ray range is demarcated by the ray, in coarse sampling, the ray range is evenly divided into a plurality of segments, one pixel point is randomly sampled in each segment, the density and the ray transmittance corresponding to each segment pixel point are recorded, coarse sampling weights are obtained, and a high weight area is determined; in fine sampling, a probability density function is constructed according to the coarse sampling weights, and new pixel points are concentrated in the high weight area by inverse transformation sampling.

[0037] Preferably, the mixed feature is inferred by a plurality of MLPs to obtain the density and the RGB color in sequence, including:

[0038] The MLP includes at least two, the mixed feature is input into the first MLP, the density is output, and the corresponding calculation formula is:

[0039]

[0040] wherein, denotes the density; denotes the activation function; denotes the mixed feature;

[0041] The mixed feature and the perspective direction are input into the second MLP, the RGB color is output, and the corresponding calculation formula is:

[0042]

[0043] wherein, denotes the RGB color; denotes the activation function; denotes the perspective direction.

[0044] Preferably, based on any image pixel of the target image, a light ray is generated in combination with the camera parameters and the camera pose, and the corresponding calculation formula is:

[0045]

[0046] wherein, represents the light ray; represents the starting point of the light ray; represents the direction of the camera optical center pointing to the image pixel corresponding to the direction of the three-dimensional space point; represents the position of the sampling point on the control light ray, that is, when represents the starting point of the light ray , when represents the direction in which the light ray extends; represents the rotation matrix of the camera.

[0047] Preferably, the multi-view texture map is rendered in combination with the probability density function and the RGB color, and the Poisson reconstruction is used to fuse the output three-dimensional model, including:

[0048] The pixel color is calculated by the discrete body rendering integral, and the corresponding calculation formula is:

[0049]

[0050]

[0051]

[0052]

[0053] wherein, represents the pixel color; represents the total number of sampling points; represents the light transmittance of the th sampling point; represents the density of the th sampling point voxel; represents the RGB color of the th sampling point; represents the density of the th sampling point; represents the interval between adjacent sampling points;

[0054] The color contribution of each sampling point is determined, the absorption and scattering processes of the light ray are simulated, the multi-view texture map is rendered, and the Poisson reconstruction is used to fuse the output three-dimensional model.

[0055] ​To solve the above problems, the application also provides a few-shot target detection system based on neural radiance fields, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the few-shot target detection method based on neural radiance fields according to any one of the preceding embodiments when executing the computer program.

[0056] The application has the following beneficial effects:

[0057] 1. The few-shot target detection method proposed in the application is used for identifying unmanned aerial vehicles, selects neural radiance fields, obtains a reconstructed three-dimensional model for target detection through the cooperative mechanism of explicit three-dimensional reconstruction and implicit feature coding, breaks through the perspective constraint of traditional two-dimensional detection, solves the model generalization bottleneck problem, can improve the accuracy of target detection even in the case of limited sample data, and has smaller data volume, higher training efficiency and faster running speed compared with conventional neural radiance fields, and is suitable for application scenarios with high real-time requirements.

[0058] 2. The application also provides a few-shot target detection system based on neural radiance fields, which is used to implement the few-shot target detection method based on neural radiance fields provided in the preceding embodiments, and has the same beneficial effects as the few-shot target detection method based on neural radiance fields in the preceding embodiments, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0060] Figure 1 The effect comparison graph of the few-shot target detection method based on neural radiance fields provided by an embodiment of the application after modeling;

[0061] Figure 2 The F1 curve graph of the verification set constructed by different methods of the few-shot target detection method based on neural radiance fields provided by an embodiment of the application;

[0062] Figure 3 The PC curve graph of the verification set constructed by different methods of the few-shot target detection method based on neural radiance fields provided by an embodiment of the application;

[0063] Figure 4A PR curve diagram of a test set constructed by different methods of a neural radiation field-based few-shot target detection method provided by an embodiment of the present application;

[0064] Figure 5 An F1 curve diagram of a test set constructed by different methods of a neural radiation field-based few-shot target detection method provided by an embodiment of the present application;

[0065] Figure 6 A PC curve diagram of a test set constructed by different methods of a neural radiation field-based few-shot target detection method provided by an embodiment of the present application;

[0066] Figure 7 A PR curve diagram of a test set constructed by different methods of a neural radiation field-based few-shot target detection method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to further illustrate the technical means and effects taken by the present application to achieve the predetermined purposes, the following describes in detail the specific embodiments, structures, features and effects of a neural radiation field-based few-shot target detection method and system according to the present application, with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0069] The specific scheme of the neural radiation field-based few-shot target detection method and system provided by the present application is specifically described below with reference to the accompanying drawings.

[0070] The first embodiment of the present application provides a neural radiation field-based few-shot target detection method, which comprises:

[0071] Step S1: Collecting a to-be-modeled object and processing to construct a target data set, obtaining the camera pose and three-dimensional coordinates of any target image in the target data set, and outputting the camera parameters;

[0072] Step S2: Setting a multi-resolution hash mapping through the three-dimensional coordinates, establishing a hash table, extracting features based on the hash table, and using trilinear interpolation to perform linear fusion to obtain mixed features;

[0073] Step S3: neural network inference of the mixed features using multiple MLPs to obtain density and RGB color, generating a light ray based on the camera pose and camera parameters, defining a light ray range based on the light ray, determining a sampling method to construct a probability density function;

[0074] Step S4: rendering the multi-view texture map by combining the probability density function and the RGB color, and using Poisson reconstruction to fuse and output a three-dimensional model;

[0075] Step S5: constructing two-dimensional images at different angles and generating a training data set, training the three-dimensional model using the training data set, obtaining a sample to be detected, and inputting it into the trained three-dimensional model to obtain a detection result.

[0076] For better illustration, the neural radiance field (NeRF) is a three-dimensional scene representation model constructed using deep learning technology. It maps each point in a three-dimensional space to a continuous function that can generate color and density information for that point, enabling precise modeling of complex three-dimensional scenes. By improving it, a small amount of sample data is used to accurately identify and locate target objects through feature extraction and representation capabilities, maintaining high detection accuracy under complex conditions such as occlusion and lighting changes. Preferably, in this embodiment, the target detection method is mainly used for unmanned aerial vehicle recognition.

[0077] Further, in step S1, it includes:

[0078] Step S11: capturing the object to be modeled to obtain a video, extracting key frames of the video through FFmpeg, and removing blurred frames through Laplacian variance filtering to construct a target data set.

[0079] Optionally, a consumer-level mobile phone is used to capture the object to be modeled, i.e., an unmanned aerial vehicle, to obtain a video. FFmpeg (Fast Forward Moving Picture Experts Group) is used to extract key frames of the video. FFmpeg is an open-source multimedia processing tool that supports multiple formats and provides a command-line interface, enabling efficient extraction of key frames from videos to provide high-quality reference images for subsequent modeling. The sampling rate is set to 2fps. Then, Laplacian variance filtering is used to remove blurred frames. The Laplacian operator is used to detect edges and details in the image, identifying unclear frames caused by motion blur or inaccurate focusing, effectively selecting frames with higher clarity and improving the overall quality of the images constructed by the modeled object.

[0080] Step S12: calling the SIFT module to extract local feature points from the target data set, generating feature descriptors, and establishing a correspondence between the feature points.

[0081] Specifically, the SIFT (Scale-Invariant Feature Transform) module of COLMAP is called to extract local feature points from the target data set, wherein in the embodiment, the peak threshold is set to 0.006, the edge threshold is 10, the key points are detected through multi-scale space, and the feature points are ensured to have high robustness and uniqueness; a 128-dimensional feature descriptor is generated, which contains local image information and can effectively distinguish different feature points; and an exhaustive matching is established to establish the corresponding relationship of the feature points, that is, all possible feature point pairs are matched to ensure the comprehensiveness and accuracy of the matching.

[0082] Step S13: obtaining an essential matrix, filtering the corresponding relationship of the feature points, performing incremental SFM reconstruction, sequentially adding camera poses and triangulating feature points to obtain camera poses and three-dimensional coordinates of any target image in the target data set, and outputting sparse point cloud and camera parameters.

[0083] Specifically, the RANSAC (Random Sample Consensus) algorithm is used to estimate the essential matrix and effectively filter the mismatch to ensure the accuracy of the target data set; then, incremental SFM (Incremental Structure from Motion) reconstruction is performed, that is, from motion reconstruction, camera poses are gradually added and feature points are triangulated to obtain camera poses and three-dimensional coordinates of the target image, and sparse point cloud and camera parameters are output.

[0084] Understandably, the multi-resolution hash mapping technology is used based on the processed target data set to improve the speed of data processing and reduce the occupation of storage space according to the multi-level mapping mode, and the multi-resolution hash mapping technology has good scalability and better performance for image processing, which can ensure efficient retrieval of data, that is, in low resolution, data in the approximate range can be quickly screened out; and in high resolution, specific data points can be accurately matched.

[0085] Further, in step S2, the following is included:

[0086] Step S21: calculating a hash index through three-dimensional coordinates, establishing a resolution network, and determining a hash table.

[0087] Further, in step S21, the following is included:

[0088] Step S211: calculating a hash index through three-dimensional coordinates, and the corresponding calculation formula is:

[0089]

[0090] wherein, represents a hash index; represents a bitwise XOR; represents the i-th dimension in a three-dimensional coordinate; represents a large prime number, i.e., a prime number assigned to the i-th dimension; represents a hash table size;

[0091] Step S212: A plurality of layer resolution networks are set, the resolution of each layer is sorted in ascending order according to the equal ratio series growth amount, the minimum resolution and the maximum resolution are determined respectively, the hierarchical index of the current layer is calculated, and a hash table is established. The corresponding calculation formula is:

[0092]

[0093] wherein, represents the hierarchical index of the i-th layer; represents the minimum resolution; represents the maximum resolution; represents the total number of layers of the resolution network; represents the i-th layer in the resolution network. As an optional implementation, the resolution network can be specifically set according to different objects to be modeled, i.e., The more layers it has, the more memory space it occupies. In the embodiment, the resolution network has 16 layers, i.e.,

[0094] ; and then arranged according to the equal ratio series growth amount, the minimum resolution , and the maximum resolution , wherein the low resolution is used to capture the global structure, and the high resolution is used to capture the local details. It can be explained that each layer dimension, hash table size and hash table item feature value are adjusted in real time according to different objects to be modeled, wherein, , the variance range of the random normal distribution is ; in the embodiment, the grid feature dimension of each layer is , the hash table size is , and the initial hash table item feature value is a random normal distribution N(0, 0.01). Step S22: Defining any pixel point in each layer of the resolution network as a target point, normalizing the three-dimensional coordinates, and locating a plurality of voxel vertices surrounding the target point through the hash table.

[0095] Step S22: Defining any pixel point in each layer of the resolution network as a target point, normalizing the three-dimensional coordinates, and locating a plurality of voxel vertices surrounding the target point through the hash table.

[0096] ​​​​It is explained that, in actual application, the target point is a pixel point to be mapped by the object to be modeled, i.e., a feature point reflecting the object to be modeled; the normalized three-dimensional coordinates are normalized to range, and then eight voxel vertices surrounding the target point are found for each layer of the hash table grid to construct a cube that precisely surrounds the target point, which is conducive to fine modeling of the object.

[0097] Step S23: obtaining the multi-dimensional feature vector of each voxel vertex from the hash table based on the hash index; that is, obtaining the multi-dimensional feature vector of each voxel vertex from the hash table based on the hash index in step S211, which contains geometric information and texture details; that is, based on the hash table, fast searching is performed to ensure the efficiency of data access, making the entire processing flow more smooth and rapid.

[0098] Step S24: determining the features of the target point by trilinear interpolation on the hash table, and splicing to obtain the mixed features.

[0099] It can be explained that trilinear interpolation is a method of data interpolation in three-dimensional space, which estimates the function value of an unknown point by calculating the function value of a known grid point, i.e., using the voxel vertex to determine the interpolation result of the target point, to smoothly transition the pixel value, reduce the sawtooth effect, and generate a more realistic three-dimensional model surface texture.

[0100] Further, in step S24, it includes:

[0101] Step S241: calculating the corresponding formula by trilinear interpolation:

[0102]

[0103] wherein, denotes the normalized target point after interpolation; denotes the position data corresponding to the normalized three-dimensional coordinates, respectively; is the weight of trilinear interpolation; denotes the normalized target point before interpolation;

[0104] Step S242: obtaining the features of all layers of the resolution network and splicing to obtain the mixed features, and the corresponding calculation formula is:

[0105]

[0106] wherein, denotes the mixed features; denotes the features after interpolation of the first layer of the resolution network; denotes the features of the last layer of the resolution network. Features after layer interpolation; Indicates the total number of layers in the resolution network.

[0107] Make an explanation, is the weight of trilinear interpolation, which is related to the target point The distance from the coordinates to the vertex is inversely proportional; since the resolution network in this embodiment is set to 16 layers, after splicing, a 32-dimensional mixed feature is obtained .

[0108] Furthermore, step S3 includes:

[0109] Step S31: Use multiple MLPs (Multi-Layer Perceptrons) to perform neural network inference on the mixed features to obtain density and RGB color in turn.

[0110] It can be shown that neural network inference of mixed features based on multi-layer perceptrons can efficiently process various input data, extract key features, and perform deep learning in each layer to ensure the high accuracy of the final output.

[0111] Furthermore, step S31 includes:

[0112] Step S311: The MLP includes at least two. The mixed features are input into the first MLP and the density is output. The corresponding calculation formula is:

[0113]

[0114] in, Indicates density; represents the activation function; Indicates mixed features;

[0115] Step S312: Input the mixed features and the viewing direction into the second MLP and output the RGB color. The corresponding calculation formula is:

[0116]

[0117] in, Represents RGB color; represents the activation function; Indicates the viewing direction.

[0118] To explain, the viewing angle direction refers to the direction of the camera light, which is converted to the spherical coordinate system and recorded as , is the pitch angle, is the azimuth, and normalized to ; preferably, in the present embodiment, the MLP is set to 3 layers, in which the width of the hidden layer is 64; that is, the mixed features are first input into the first MLP, and the output density is output; then the mixed features and the view direction are spliced, and input into the second MLP, and the RGB color is output.

[0119] Step S32: generating a light ray based on any image pixel of the target image, in combination with the camera parameter and the camera pose.

[0120] Further, in step S32, the corresponding calculation formula is:

[0121]

[0122] wherein, represents the light ray; represents the starting point of the light ray; represents the direction of the camera optical center pointing to the image pixel corresponding to the direction of the three-dimensional space point; represents the position of the sampling point on the control light ray, that is, when represents the starting point of the light ray , when , represents that the light ray extends in the direction; represents the rotation matrix of the camera.

[0123] It can be explained that the light ray is generated, that is, each pixel point is mapped to a specific position in the three-dimensional space, forming a light ray starting from the camera optical center and passing through the pixel point; wherein the camera pose involves the position and direction of the camera in the three-dimensional space, in combination with the camera parameter, the direction and path of the light ray are determined, and the accurate capture and reconstruction of the modeled object are realized.

[0124] Step S33: defining the light ray range by the light ray, in the coarse sampling, the light ray range is evenly divided into several segments, and a pixel point is randomly sampled in each segment, the density and light ray transmittance of the corresponding pixel point in each segment are recorded, the coarse sampling weight is obtained, and the high weight area is determined; in the fine sampling, the probability density function is constructed according to the coarse sampling weight, and the newly added pixel points are concentrated in the high weight area by inverse transformation sampling.

[0125] Specifically, in the present embodiment, the light ray range is evenly divided into 64 segments; in the coarse sampling stage, a pixel point is randomly sampled in each of the 64 segments, the density and light ray transmittance of each pixel point are recorded, and the coarse sampling weight is determined as , wherein, represents the weight of the th sampling point, that is, the th pixel point in the coarse sampling; represents the weight of the The light transmittance of the i-th pixel point; The density of the i-th pixel point; The sampling interval between the i-th pixel point and its adjacent pixel point; to construct a probability density function, and new 128 points are concentrated in the high weight area by inverse transform sampling. Further, in step S4, the following steps are included:

[0126] Step S41: calculate the pixel color by discrete volume rendering integral, and the corresponding calculation formula is:

[0127]

[0128]

[0129]

[0130]

[0131]

[0132] Among them, The pixel color is represented by C; The total number of sampling points is represented by N; The light transmittance of the i-th sampling point; The density of the i-th sampling point voxel; The RGB color of the i-th sampling point; The density of the i-th sampling point; The interval between adjacent sampling points is represented by D; Step S42: determine the color contribution of each sampling point, simulate the absorption and scattering process of light, render a multi-view texture map, and use Poisson reconstruction to fuse and output a three-dimensional model. It can be understood that the multi-view texture map can exhibit rich color levels and details under different lighting conditions, enhancing the visual effect. Then, using the Poisson reconstruction technology, the multi-view texture map is seamlessly fused to construct a high-precision three-dimensional model, ensuring smooth transition and realism of the model performance, so that the final three-dimensional model has high geometric accuracy.

[0133] Further, in step S5, construct two-dimensional images of different angles and generate a training data set, train the three-dimensional model using the training data set, obtain the sample to be detected, and input it into the trained three-dimensional model to obtain the detection result.

[0134] It can be understood that the multi-view texture map can exhibit rich color levels and details under different lighting conditions, enhancing the visual effect. Then, using the Poisson reconstruction technology, the multi-view texture map is seamlessly fused to construct a high-precision three-dimensional model, ensuring smooth transition and realism of the model performance, so that the final three-dimensional model has high geometric accuracy.

[0135] Further, in step S5, construct two-dimensional images of different angles and generate a training data set, train the three-dimensional model using the training data set, obtain the sample to be detected, and input it into the trained three-dimensional model to obtain the detection result.

[0136] ​​​​Specifically, download the BlenderNerf plugin and install it to Blender, set the training frame number to 100 frames and the camera track after enabling the plugin, ensure that the camera path is smooth and covers all key perspectives; import the three-dimensional model, control the camera angle through script loop, generate two-dimensional images of different angles, that is, 1253 multi-perspective two-dimensional images and save them as sequence frames in an orderly manner; then use Labelimg manual annotation to change the newly generated two-dimensional images into yolo format, and add the original 1955 drone photos to form a new drone NerfedData dataset, that is, a training dataset composed of 2645 training sets, 568 validation sets and 560 test sets; and then train the three-dimensional model.

[0137] For reference Figure 1 To ensure the universality of the recognition effect, the most common yolov5-5.0 is selected as the model framework, the batch-size is selected as 4, and the training period is set to 200 rounds; after the training is completed, the accuracy of the validation set is 93%, the recall rate is 86.3%, the mAP@0.5 index is 0.893, and the mAP@.5:.95 index is 0.575; the accuracy of the test set is 90.8%, the recall rate is 79.8%, the mAP@.5 is 0.84, and the mAP@.5:.95 is 0.462; and it can be seen that the modeling image is clear, and the effect reaches the level required by the task.

[0138] Understandably, the few-shot target detection method proposed in the application identifies the unmanned aerial vehicle, selects the neural radiation field, obtains the reconstructed three-dimensional model for target detection through the cooperative mechanism of explicit three-dimensional reconstruction and implicit feature coding, breaks through the perspective constraint of traditional two-dimensional detection, solves the model generalization bottleneck problem, and even in the case of limited sample data, it can also improve the accuracy of target detection, and compared with the conventional neural radiation field, the data volume is small, the training is more efficient, the running speed is faster, and it is suitable for application scenarios with high real-time requirements.

[0139] For reference Figures 2-7For better illustration, as a comparison, the target data set is defined as the original data set, a new originAug data set is generated based on the original data set through three times of data augmentation in the form of flip rotation, etc., and based on the yolov5-5.0 framework, the “airplane” and “bird” categories under the coco data set are learned as the pre-training weight of transfer learning to perform the recognition training of the unmanned aerial vehicle; the training data set is the Nerfed data set, three data are respectively recognized and trained, and the verification set and the test set in each data set are respectively compared to obtain the F1 curve, the PC (Precision-Confidence, i.e., precision confidence) curve and the PR (Precision-Recall, i.e., precision recall) curve in turn, wherein the F1 curve represents the relationship between the F1 value and the confidence to evaluate the performance of the model for the data sets constructed after the data processing by different methods; the PC curve shows the relationship between the confidence of the model prediction and its actual precision; the PR curve further reveals the performance of the model under different conditions by showing the change of the precision of the model under different recall rates; to provide intuitive data analysis.

[0140] It can be explained that for the verification set of the three data sets, in the F1 curve, each curve is labeled with the corresponding F1 value and the value under any confidence threshold, wherein the F1 value of the Nerfed data set is relatively high in most confidence intervals, and the highest F1 value is 0.86, and the corresponding confidence threshold is 0.514; it is explained that the Nerfed data set processed by the target detection method of the present application can achieve a good performance balance on the whole, and performs outstandingly in improving the model performance; in the PC curve, the Nerfed data set has a faster precision rising speed in the confidence interval with lower confidence, and the precision reaches 1.0 when the confidence is 0.868, which indicates that the Nerfed data set can enable the model to achieve perfect precision at a relatively high confidence, and the performance is relatively good; in the PR curve, the Nerfed data set has obvious advantages in precision-recall, which can better meet the needs of practical applications, and thus provide more reliable target detection results, and the effect is better.

[0141] In the F1 curve, the Nerfed dataset performs best in the relationship between F1 value and confidence, which can obtain the highest F1 value at a suitable confidence threshold, indicating that it performs best in comprehensive performance; in the PC curve, the position of the Nerfed dataset is generally higher than the other two curves in the entire confidence interval, and only at very low confidence, the three curves are close, indicating that the Nerfed dataset can provide higher precision at the same confidence level, so the Nerfed dataset is superior to the other two in terms of confidence requirements for high precision, overall precision performance, model reliability and stability, etc.; in the PR curve, the average precision (mAP) of the three different datasets at the IoU threshold of 0.5 is marked, the curve corresponding to the Nerfed dataset still maintains a high precision at a low recall rate, and with the increase of the recall rate, the precision gradually decreases, but the decrease speed is relatively gentle, from the mAP value, the mAP of the Nerfed dataset is the highest, which is 0.840, indicating that it has the best comprehensive performance in the target detection task; from the recall rate, it can still maintain a relatively high precision at a high recall rate, indicating that the Nerfed dataset performs well in balancing precision and recall; it can be further explained that the Nerfed dataset performs best in the target detection task, can maintain a high precision at different recall levels, and has the highest mAP value

[0142] Then, taking accuracy, recall rate, mAP@0.5 and mAP@.5:.95 as performance evaluation indexes, Tables 1 and 2 are obtained respectively, as shown in the following table:

[0143] Table 1 Comparison table of effects of validation sets of different datasets

[0144]

[0145] Table 2 Comparison table of effects of test sets of different datasets

[0146]

[0147] It can be explained that the Nerfed dataset performs best in improving model accuracy, mAP@.5 and mAP@.5:.95, indicating that this dataset has the best training effect on the model; that is, the recognition effect after neural radiation field modeling is much better than that of the original dataset and the dataset after data enhancement and transfer learning.

[0148] The second embodiment of the present application also provides a few-shot target detection system based on neural radiance fields, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the few-shot target detection method based on neural radiance fields according to any one of the preceding embodiments.

[0149] It should be noted that the few-shot target detection system based on neural radiance fields has the same beneficial effects as the few-shot target detection method based on neural radiance fields described above, and will not be repeated here.

[0150] It should be noted that the above-mentioned embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or may be advantageous.

[0151] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment mainly describes the difference from other embodiments.

Claims

1. A few-shot target detection method based on neural radiation field, characterized in that: The method comprises: Collect the objects to be modeled and process them to construct the target data set, obtain the camera pose and 3D coordinates of any target image in the target data set, and output the camera parameters; Multi-resolution hash mapping is set by three-dimensional coordinates, a hash table is established, features are extracted based on the hash table, and trilinear interpolation is used for linear fusion to obtain hybrid features; Use multiple MLPs to perform neural network inference on mixed features to obtain density and RGB color. Generate light through camera pose and camera parameters. Determine the light range based on the light, determine the sampling method and construct the probability density function. Combining the probability density function and RGB color for rendering to obtain a multi-view texture map, and using Poisson reconstruction fusion to output a 3D model; Construct two-dimensional images from different angles and generate a training data set. Use the training data set to train a three-dimensional model, obtain samples to be tested, and input them into the trained three-dimensional model to obtain the detection results.

2. The method for detecting a small number of targets based on neural radiation fields according to claim 1, wherein: Collect the objects to be modeled and process them to construct a target dataset, and obtain the camera pose and 3D coordinates of any target image in the target dataset, including: Shoot the object to be modeled to obtain a video, extract the key frames of the video using FFmpeg, and remove the blurred frames through Laplacian variance filtering to construct the target dataset; Call the SIFT module to extract local feature points from the target data set, generate feature descriptors, and establish feature point correspondences; Obtain the essential matrix, filter the correspondence of feature points, perform incremental SFM reconstruction, add camera poses and triangulate feature points in sequence, obtain the camera pose and 3D coordinates of any target image in the target dataset, and output a sparse point cloud and camera parameters.

3. The method for detecting a small number of targets based on neural radiation fields according to claim 1, wherein: Multi-resolution hash mapping is set by three-dimensional coordinates, a hash table is established, features are extracted based on the hash table, and trilinear interpolation is used for linear fusion to obtain hybrid features, including: Calculate hash index through three-dimensional coordinates, build resolution network, and determine hash table; Define any pixel point in each layer of the resolution network as the target point, normalize the three-dimensional coordinates, and locate multiple voxel vertices surrounding the target point through a hash table; Obtain a multi-dimensional feature vector from the hash table for each voxel vertex based on the hash index; The features of the target point are determined by trilinear interpolation of the hash table, and then spliced ​​to obtain the mixed features.

4. The method for detecting a small number of targets based on neural radiation fields according to claim 3, wherein: Calculate the hash index through the three-dimensional coordinates, establish the resolution network, and determine the hash table, including: The hash index is calculated by three-dimensional coordinates. The corresponding calculation formula is: ； in, Represents a hash index; Represents bitwise exclusive OR; Represents the first dimensions; Represents a large prime number, that is, The allocated prime number of dimensions; Indicates the size of the hash table; Set up several layers of resolution networks, sort the resolution of each layer in ascending order according to the geometric progression, determine the minimum resolution and maximum resolution, calculate the level index of the current layer, and build a hash table. The corresponding calculation formula is: ； in, Indicates the The level index of the layer; Indicates the minimum resolution; Indicates the maximum resolution; Indicates the total number of layers in the resolution network; Represents the first layer.

5. The method for detecting a small number of targets based on neural radiation fields according to claim 4, wherein: The hash table is used to determine the characteristics of the target point through trilinear interpolation, and then spliced ​​to obtain mixed features, including: Through trilinear interpolation, the corresponding calculation formula is: ； in, Represents the current normalized target point Features after interpolation; Respectively represent the position data corresponding to the normalized three-dimensional coordinates; is the weight of trilinear interpolation; Represents the current normalized target point Features before interpolation; Get the features of all layers of the resolution network and concatenate them to get the mixed features. The corresponding calculation formula is: ； in, Indicates mixed features; Represents the features after interpolation of the first layer of the resolution network; Represents the resolution network Features after layer interpolation; Indicates the total number of layers in the resolution network.

6. The method for detecting a small number of targets based on neural radiation fields according to claim 1, wherein: Multiple MLPs are used to perform neural network inference on mixed features to obtain density and RGB color. Light is generated by combining camera pose and camera parameters. The light range is defined based on the light, and the sampling method is determined to construct a probability density function, including: Use multiple MLPs to perform neural network inference on the mixed features to obtain density and RGB color in turn; Generate light based on any image pixel of the target image, combined with camera parameters and camera pose; The light range is defined by light. During coarse sampling, the light range is evenly divided into several segments. A pixel point is randomly sampled in each segment. The density and light transmittance of the pixel points corresponding to each segment are recorded to obtain the coarse sampling weight and determine the high-weight area. During fine sampling, a probability density function is constructed according to the coarse sampling weight, and the newly added pixel points are concentrated in the high-weight area through inverse transformation sampling.

7. The method for detecting a small number of targets based on neural radiation fields according to claim 6, wherein: Using multiple MLPs to perform neural network inference on mixed features, density and RGB color are obtained in turn, including: MLP consists of at least two, the mixed features are input into the first MLP, and the output density is calculated as follows: ； in, Indicates density; represents the activation function; Indicates mixed features; The mixed features and viewing direction are input into the second MLP, and the RGB color is output. The corresponding calculation formula is: ； in, Represents RGB color; represents the activation function; Indicates the viewing direction.

8. The method for detecting a small number of targets based on neural radiation fields according to claim 6, wherein: Based on any image pixel of the target image, combined with the camera parameters and camera pose, light is generated. The corresponding calculation formula is: ； in, Indicates light; Indicates the starting point of the light; Indicates that the camera optical center points to the image pixel The direction corresponding to the point in three-dimensional space; Indicates the position of the sampling point on the control ray, that is, when When , it indicates the starting point of the light ,when When Direction extension; Represents the camera's rotation matrix.

9. The method for detecting a small number of targets based on neural radiation fields according to claim 1, wherein: Combining the probability density function and RGB color for rendering to obtain a multi-view texture map, and using Poisson reconstruction to fuse the output 3D model, including: The pixel color is calculated by discrete volume rendering integral, and the corresponding calculation formula is: ； ； ； ； in, Indicates pixel color; Indicates the total number of sampling points; Indicates the Light transmittance of each sampling point; Indicates the The density of the voxels at each sampling point; Indicates the The RGB color of the sampling point; Indicates the The density of sampling points; Indicates the interval between adjacent sampling points; Determine the color contribution of each sampling point, simulate the absorption and scattering process of light, render a multi-view texture map, and use Poisson reconstruction to fuse the output 3D model.

10. A few-shot target detection system based on neural radiation fields, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a few-sample target detection method based on neural radiation field are implemented as described in any one of claims 1 to 9.

Images