4D space-time field construction method and device based on neural field reconstruction

Abstract

The invention provides a 4D space-time field construction method and device based on neural field reconstruction, and relates to the technical field of data processing. A 4D space-time field construction method based on neural field reconstruction comprises the following steps: performing supervision information labeling on a plurality of three-dimensional coordinates in a to-be-processed 3D model to obtain a plurality of training data samples so as to construct a training data set; the three-dimensional space is divided into a plurality of cubic grids of the same size, a neural network corresponds to the interior of each grid, and therefore a block neural field model is constructed; training a block neural field model based on the training data set to obtain a target 4D space-time field; and in response to local data updating of the 3D model to be processed, updating the target 4D space-time field. According to the method, dynamic updating of the 4D space-time field is converted into an incremental reasoning problem of a neural network through block neural field modeling and a continuous learning mechanism, and efficient updating of the 4D space-time field is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a 4D space-time field construction method and device based on neural field reconstruction. BACKGROUND

[0002] Currently, in a low-altitude operation system, a 4D space-time field needs to express three-dimensional spatial geometric information (such as terrain, buildings, and obstacles) and dynamic changes in the time dimension (such as weather updates and control area adjustments).

[0003] Existing technologies mostly directly store 3D models (such as point clouds and meshes) and rely on special databases to achieve dynamic updates. However, the dependence on explicit 3D models (such as meshes and voxels) makes it difficult to cope with the efficient update requirements of dynamic environments (such as temporary control areas and sudden weather changes). SUMMARY

[0004] Therefore, the present application provides a 4D space-time field construction method and device based on neural field reconstruction, which solves the problem of dependence on explicit 3D models in the prior art and realizes the construction of a 4D space-time field that can be efficiently updated.

[0005] According to an aspect of the present application, a 4D space-time field construction method based on neural field reconstruction is provided, which includes: labeling supervision information for a plurality of three-dimensional coordinates in a to-be-processed 3D model to obtain a plurality of training data samples, thereby constructing a training data set; dividing a three-dimensional space into a plurality of cubic grids of the same size, each grid corresponding to a neural network inside, thereby constructing a block neural field model; training the block neural field model based on the training data set to obtain a target 4D space-time field; and updating the target 4D space-time field in response to local data updates of the to-be-processed 3D model.

[0006] According to some embodiments, training the block neural field model based on the training data set to obtain the target 4D space-time field includes: obtaining time-varying data of the plurality of training data samples in the training data set within a preset time range and their timestamps, and obtaining a plurality of four-dimensional training data samples according to the plurality of training data samples and their timestamps; encoding the time-varying data to obtain time-varying vectors; and training the block neural field model according to the plurality of four-dimensional training data samples and their time-varying vectors using a pre-constructed hybrid loss function until a preset end condition is reached, and outputting the training result as the target 4D space-time field.

[0007] According to some embodiments, a pre-constructed hybrid loss function is utilized to train the block neural field model according to a plurality of four-dimensional training data samples and time-varying vectors thereof until a preset ending condition is reached, and a training result is output as a target 4D spatio-temporal field, comprising: extracting a query vector from the plurality of four-dimensional training data samples; utilizing a preset attention mechanism to obtain weighted conditional features corresponding to the plurality of four-dimensional training data samples according to the query vector and the time-varying vectors corresponding thereto; utilizing the pre-constructed hybrid loss function to train the block neural field model according to the plurality of four-dimensional training data samples and the weighted conditional features thereof until the preset ending condition is reached, and outputting a training result as the target 4D spatio-temporal field.

[0008] According to some embodiments, the target 4D spatio-temporal field is updated in response to local data updates of the 3D model to be processed, comprising: in response to the local data updates of the 3D model to be processed, obtaining one or more neural networks in the target 4D spatio-temporal field corresponding to the update data as target neural networks; constructing a local training data set according to the update data, and fine-tuning the target neural networks based on the local training data set to obtain an updated target 4D spatio-temporal field.

[0009] According to some embodiments, a plurality of three-dimensional coordinates in the 3D model to be processed are labeled with supervision information to obtain a plurality of training data samples, thereby constructing a training data set, comprising: data augmentation is performed on the training data set, and the training data set is updated according to the results of the data augmentation.

[0010] According to some embodiments, the hybrid loss function comprises:

[0011] L = L sdf + λL cls ;

[0012]

[0013] wherein L is a hybrid loss function, L sdf is an SDF regression loss function, L cls is a semantic classification loss function, λ is a weight coefficient, x i is the coordinate of a spatial point, y c is the true value of the semantic classification class, N is the total number of SDF points, is the estimated value of the semantic classification class, c is the total number of semantic classification classes, f is an SDF calculation function, and φ is a network alias for estimating SDF.

[0014] According to some embodiments, the method further includes: obtaining time-varying data of the plurality of training data samples in the training data set within a preset time range and time stamps thereof, including: obtaining discrete time-varying information of the plurality of training data samples in the training data set; and diffusing the discrete time-varying information into time-varying data at a plurality of time stamps based on a pre-constructed time-varying information prediction model.

[0015] According to some embodiments, the method further includes: constructing a path data set according to the target 4D space-time field, wherein the path data set includes a plurality of path data samples, and each path data sample includes a start point sample, an end point sample, and path information samples from the start point sample to the end point sample; training a path planning model based on the path data set; and performing path planning based on the path planning model.

[0016] According to some embodiments, the method further includes: constructing a path data set according to the target 4D space-time field, including: S1: obtaining occupancy information of a plurality of three-dimensional coordinates according to the target 4D space-time field; S2: selecting a start point sample and an end point sample from the plurality of three-dimensional coordinates; S3: determining a path from the start point sample to the end point sample according to the occupancy information, and taking the path as path information samples corresponding to the start point sample and the end point sample, thereby obtaining a path data sample; and S4: repeating steps S2-S3 to obtain the path data set.

[0017] According to some embodiments, the method further includes: distilling the path planning model.

[0018] According to some embodiments, the method further includes: updating the path planning model in response to an update of the target 4D space-time field.

[0019] According to an aspect of the present application, a 4D space-time field construction device based on neural field reconstruction includes: a data set construction unit configured to label a plurality of three-dimensional coordinates in a to-be-processed 3D model with supervision information to obtain a plurality of training data samples, thereby constructing a training data set; a neural field construction unit configured to divide a three-dimensional space into a plurality of cubic grids of the same size, each grid corresponding to a neural network inside, thereby constructing a block neural field model; a space-time field training unit configured to train the block neural field model based on the training data set to obtain a target 4D space-time field; and a space-time field updating unit configured to update the target 4D space-time field in response to local data update of the to-be-processed 3D model.

[0020] According to an aspect of the present application, an electronic device is provided, which includes: one or more processors; a storage device configured to store one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0021] According to an aspect of the present application, a computer readable medium is provided, which stores a computer program or instructions, and the program or instructions are executed by a processor to implement the method as described above.

[0022] Through the above embodiments provided by the present application, the plurality of three-dimensional coordinates in the to-be-processed 3D model are supervisedly labeled to construct a training data set; the three-dimensional space is divided into a plurality of cubic grids, so that each grid corresponds to a neural network inside, a block neural field model is constructed, the block neural field model is trained based on the training data set, and a target 4D space-time field is obtained. The target 4D space-time field can be fine-tuned according to local data update of the to-be-processed 3D model. The present application converts the dynamic update of the 4D space-time field into an incremental inference problem of the neural network through the block neural field modeling and the continuous learning mechanism, and realizes efficient update of the 4D space-time field. BRIEF DESCRIPTION OF DRAWINGS

[0023] It should be understood that the above general description and the following detailed description are only exemplary and cannot limit the present application.

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art according to these drawings without departing from the scope of the present application.

[0025] Figure 1 The flowchart of the 4D space-time field construction method based on the neural field reconstruction provided by the embodiments of the present application;

[0026] Figure 2 The flowchart of training the block neural field model based on the training data set to obtain the target 4D space-time field provided by the embodiments of the present application;

[0027] Figure 3 The flowchart of training the block neural field model based on the training data set to obtain the target 4D space-time field provided by the embodiments of the present application;

[0028] Figure 4 The flowchart of updating the target 4D space-time field in response to the local data update of the to-be-processed 3D model provided by the embodiments of the present application;

[0029] Figure 5 The flowchart of obtaining the time-varying data and time stamp of the plurality of training data samples in the training data set within a preset time range provided by the embodiments of the present application;

[0030] Figure 6 A flowchart of constructing a path dataset according to a target 4D space-time field is provided for the embodiments of the present application;

[0031] Figure 7 A block diagram of a 4D space-time field construction device based on neural field reconstruction is provided for the embodiments of the present application;

[0032] Figure 8 A structural schematic diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0034] In addition, the described features, structures or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a sufficient understanding of the embodiments of the present application. However, those skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or can employ other methods, components, devices, steps, etc. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring the aspects of the present application.

[0035] The block diagrams shown in the drawings are only functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the drawings are only exemplary illustrations, which do not necessarily include all the contents and operations / steps, and are not necessarily executed in the described order. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so that the actual execution order can be changed according to the actual situation.

[0037] It should be understood that although the terms first, second, third, etc. can be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be called the second component without departing from the teachings of the present application concepts. As used herein, the term "and / or" includes any one and all combinations of the associated listed items.

[0038] The specific implementation can refer to the following embodiments.

[0039] Figure 1 A flowchart of a 4D spatiotemporal field construction method based on neural field reconstruction is provided for the embodiments of the present application. As shown in the figure, the method comprises steps S110-S140. Figure 1

[0040] In step S110, the multiple three-dimensional coordinates in the to-be-processed 3D model are labeled with supervision information to obtain multiple training data samples, thereby constructing a training data set.

[0041] The to-be-processed 3D model referred to in the present application includes the 3D model of the entity (such as terrain, building, obstacle) in the three-dimensional space in the target area. The present application does not limit the expression form, and the expression form of the to-be-processed 3D model includes point cloud, mesh, voxel for calculation, and OSGB, GLB, GLTF, 3D Tiles and the like for display. In addition, the scene and asset files commonly used in the game field (such as OBJ, STL, FBX) can also be used as the expression form.

[0042] Among them, the target area refers to the three-dimensional area related to the 4D spatiotemporal field to be constructed. It needs to be emphasized that the 4D spatiotemporal field referred to in the present application is a continuous spatiotemporal structure formed by taking time as the fourth dimension and three-dimensional space, which includes three-dimensional space geometric information and dynamic changes in the time dimension, i.e. near real-time update of three-dimensional space.

[0043] The multiple three-dimensional coordinates are extracted from the to-be-processed 3D model, and the extracted three-dimensional coordinates are labeled with supervision information to obtain multiple training data samples, thereby obtaining a training data set.

[0044] According to an example embodiment, the supervision information labeling includes at least one of symbol occupancy field labeling (SDF value labeling) and semantic label labeling.

[0045] In step S120, the three-dimensional space is divided into multiple cubic grids of the same size, and each grid corresponds to a neural network inside, thereby constructing a block neural field model.

[0046] The present application uses neural radiation field to construct 4D spatiotemporal field, which aims to improve the traditional 4D spatiotemporal field which is difficult to cope with the efficient update demand of dynamic environment.

[0047] ​It needs to be explained that the neural implicit field technology (for example, the SDF network) encodes geometric information implicitly through a neural network, has the advantages of continuous expression, dynamic updating, and lightweight, that is, the neural implicit field technology can continuously express, does not need to be discretized and stored, supports arbitrary resolution query, and can effectively improve the storage efficiency and flexibility of the traditional 4D space-time field; the neural implicit field technology can be dynamically updated, can realize local model correction through incremental training, and can meet the efficient updating demand of the dynamic environment; a single network of the neural implicit field technology can be compressed to the MB level, and is suitable for edge deployment.

[0048] Based on this, the neural implicit field technology is used to divide a three-dimensional space into grids, and each grid corresponds to a neural network. The input of the neural network is a three-dimensional coordinate (x, y, z), and the output is a supervised information label corresponding to related content.

[0049] According to an example embodiment, the supervised information label is an SDF value label and a semantic label (such as "building", "vegetation", etc.) label, and correspondingly, the output of the neural network is an SDF value and a semantic label.

[0050] The application does not limit the specific model of the neural network, which can be selected according to actual conditions.

[0051] According to an example embodiment, each three-dimensional space grid corresponds to an MLP (Multilayer Perceptron, Multilayer Perceptron).

[0052] The plurality of neural networks are superimposed according to the corresponding three-dimensional space grid to construct a block neural field model.

[0053] In step S130, the block neural field model is trained based on the training data set to obtain a target 4D space-time field.

[0054] The training data set is used to train and optimize the block neural field model, and the application does not limit the specific training algorithm, which can be selected according to actual conditions, and the training result is output as the target 4D space-time field.

[0055] In step S140, the target 4D space-time field is updated in response to local data update of the to-be-processed 3D model.

[0056] When the local area of the to-be-processed 3D model changes, only the network of the corresponding block is fine-tuned.

[0057] According to an example embodiment, a new data set is constructed according to the local data of the updated local area, and the target 4D space-time field is fine-tuned using the new data set.

[0058] Further, in some embodiments, the application further comprises: in the updating process of step S140, adopting an elastic weight consolidation (EWC) algorithm to limit the update range of key parameters to prevent catastrophic forgetting.

[0059] Based on the above embodiments, the application can achieve a network inference delay of <10ms (single NVIDIA T4 GPU) and good real-time inference capability.

[0060] The application also supports batch query (Batch Query) to improve throughput.

[0061] The application supervises and labels a plurality of three-dimensional coordinates in a to-be-processed 3D model to construct a training data set; divides a three-dimensional space into a plurality of cubic grids, so that each grid corresponds to a neural network inside, constructs a block neural field model, trains the block neural field model based on the training data set, and obtains a target 4D space-time field. The target 4D space-time field can be fine-tuned according to local data update of the to-be-processed 3D model. The application converts the dynamic update of the 4D space-time field into an incremental inference problem of the neural network through the block neural field modeling and the continuous learning mechanism, and realizes efficient update of the 4D space-time field.

[0062] According to some embodiments, referring to Figure 2 In step S130, the block neural field model is trained based on the training data set to obtain a target 4D space-time field, which can be realized by steps S210-S230.

[0063] In step S210, time-varying data of a plurality of training data samples in the training data set in a preset time range and time stamps thereof are obtained, and a plurality of four-dimensional training data samples are obtained according to the plurality of training data samples and the time stamps.

[0064] Specifically, a time stamp input (t) needs to be added to each block network to model time-varying characteristics (such as meteorological diffusion, etc.).

[0065] In actual implementation, first, time-varying data of a plurality of training data samples in the training data set in a preset time range and time stamps thereof are obtained. The preset time range refers to the sampling range, which can be selected according to actual conditions.

[0066] In order to increase the accuracy of the trained model, a time range covering time nodes with various time-varying characteristics is selected as the preset time range, and time stamps corresponding to time-varying data of time nodes with time-varying characteristics are obtained.

[0067] Time stamps are added to the training data samples to obtain four-dimensional training data samples.

[0068] In step S220, the time-varying data is encoded to obtain a time-varying vector.

[0069] According to an example embodiment, the Transformer is used to encode the time series data to capture long-term dependencies.

[0070] In step S230, the hybrid loss function is used to train the block neural field model according to the plurality of four-dimensional training data samples and the time-varying vectors thereof until a preset end condition is met, and the training result is output as a target 4D spatiotemporal field.

[0071] The hybrid loss function is used to train the block neural field model.

[0072] During the training process, the time-varying vector can be embedded in the four-dimensional training data sample in a labeled form, or the time-varying vector can be fused with the four-dimensional training data sample through an attention mechanism to participate in the training.

[0073] The preset end condition can include the number of iterations or the preset evaluation index meeting the condition, which is not limited in the present application.

[0074] When the preset end condition is met, the training result is output as a target 4D spatiotemporal field

[0075] According to some embodiments, with reference to Figure 3 In step S230, the hybrid loss function is used to train the block neural field model according to the plurality of four-dimensional training data samples and the time-varying vectors thereof until a preset end condition is met, and the training result is output as a target 4D spatiotemporal field, which can be implemented through steps S310-S330.

[0076] In step S310, a query vector is extracted from the plurality of four-dimensional training data samples.

[0077] The backbone input (x, y, z, t) is extracted as the query vector (Query).

[0078] Wherein (x, y, z) is a three-dimensional coordinate and t is a timestamp.

[0079] In step S320, a preset attention mechanism is used to obtain weighted conditional features corresponding to the plurality of four-dimensional training data samples according to the query vector and the time-varying vector corresponding thereto.

[0080] In order to fuse external time-varying conditions such as weather and temporary control area, a multi-condition fusion strategy based on attention mechanism is adopted in this embodiment. The core idea is to let the neural network dynamically focus on and weight the influence of different external conditions according to the current time and spatial position, so as to adjust the output of the neural network. The attention mechanism can determine which conditions are more important at this time and place, and give higher weight.

[0081] It should be noted that the external condition input (such as weather data, control area coordinates + time window, etc.) is time-varying data, which is pre-encoded into a time-varying vector format and enters the module related to the attention mechanism. For example, according to an example embodiment, the weather data (temperature, wind speed, humidity) is spliced into a time-varying vector v_weather; the control area information (latitude and longitude range, effective time, invalid time) is encoded into a time-varying vector v_zone; other time-varying data such as traffic and airspace restrictions can also be encoded into corresponding time-varying vectors according to the situation.

[0082] These external features are uniformly mapped through a preset attention mechanism with a neural network to obtain weighted condition features.

[0083] According to an example embodiment, a small MLP is used for feature mapping to form keys and values, and a standard scaled dot-product attention mechanism (Scaled Dot-Product Attention) is selected as the preset attention mechanism:

[0084]

[0085] Where Q is the query vector (extracted from (x, y, z, t)); K is the key vector (the representation after encoding the external condition, i.e. the time-varying vector), V: is the value vector (the strength of the feature influence of each condition), d k is a scaling factor to avoid gradient vanishing caused by large inner product.

[0086] Based on this, the attention weight is applied to the value vector to obtain the weighted and fused condition features, which are the weighted condition features.

[0087] It should be emphasized that the preset attention mechanism can be a multi-head attention mechanism, or a self-attention and single-head attention mechanism, which is not limited by the present application.

[0088] In step S330, the mixed loss function is used to train the block neural field model according to the plurality of four-dimensional training data samples and their weighted condition features until a preset end condition is reached, and the training result is output as the target 4D spatio-temporal field.

[0089] The weighted conditional features are concatenated or added to the intermediate layer outputs of the backbone network, and the subsequent network layers use the fused features to predict the final model output.

[0090] For example, in one embodiment, when a spatiotemporal point is within a time window of a certain temporary flight restriction area, the attention mechanism gives higher weight to the area. The model automatically raises the SDF output value of the area (i.e., the surface is harder), making it considered as an untraversable area in path planning.

[0091] According to some embodiments, with reference to Figure 4 In step S140, in response to the local data update of the 3D model to be processed, the target 4D spatiotemporal field is updated, which can be implemented by steps S410-S420.

[0092] In step S410, in response to the local data update of the 3D model to be processed, one or more neural networks in the target 4D spatiotemporal field corresponding to the update data are obtained as target neural networks.

[0093] After the 4D spatiotemporal field is constructed, when a local area is detected to have data update (e.g., a new building), the neural network related to the updated local area is located. Specifically, the grid related to the local area in the 3D space is located, denoted as a target grid, and one or more neural networks related to the target grid are determined as target neural networks.

[0094] In step S420, a local training data set is constructed according to the update data, and the target neural network is fine-tuned based on the local training data set to obtain an updated target 4D spatiotemporal field.

[0095] In the actual update process, a continuous learning mechanism is used to fine-tune the target neural network using the update data of the updated area to construct a new local training data set.

[0096] The present application only updates the neural network corresponding to the affected area to obtain an updated 4D spatiotemporal field, which is time-consuming and low in complexity.

[0097] According to some embodiments, in step S110, the multiple three-dimensional coordinates in the 3D model to be processed are labeled with supervision information to obtain multiple training data samples, thereby constructing a training data set, and further comprising step S111.

[0098] In step S111, the training data set is subjected to data augmentation, and the training data set is updated according to the result of data augmentation.

[0099] On the basis of the training data set constructed by the multiple training data samples, the training data samples can also be subjected to data augmentation to expand the number of training samples and improve the robustness of the model.

[0100] According to an example embodiment, the data augmentation specifically comprises adding noise (e.g. ±5% coordinate perturbation) to the training data samples.

[0101] Further, in some embodiments, based on the supervised labeling of SDF value labeling and semantic label labeling, the hybrid loss function is constructed by combining the SDF regression loss (L1 Loss) and the semantic classification loss (Cross-Entropy), and at this time, the model output is the SDF value and / or the semantic label.

[0102] Based on the above embodiments, the hybrid loss function comprises:

[0103] L = L sdf + λL cls ;

[0104]

[0105] wherein L is the hybrid loss function, L sdf is the SDF regression loss function, L cls is the semantic classification loss function, λ is the weight coefficient, x i is the coordinate of the spatial point, y c is the true value of the semantic classification class, N is the total number of SDF points calculated, is the estimated value of the semantic classification class, c is the total number of semantic classification classes, f is the SDF calculation function, and φ is the network for estimating SDF. It needs to be noted that the semantic classification class is the semantic label.

[0106] According to some embodiments, with reference to Figure 5 In step S210, the time-varying data of the plurality of training data samples in the training data set within a preset time range and the time stamp thereof are obtained, which can be implemented by steps S510-S520.

[0107] In step S510, the discrete time-varying information of the plurality of training data samples in the training data set is obtained.

[0108] It needs to be noted that part of the time-varying data can be in a discrete state, such as minute-level or hour-level meteorological data.

[0109] On this basis, the discrete time-varying information needs to be predicted and filled to obtain continuous time-varying information.

[0110] Firstly, the discrete time-varying information needs to be obtained.

[0111] It should be emphasized that the purpose of step S210 is to obtain time-varying data in a preset time range, but since the present embodiment is to predict continuous time-varying data from discrete time-varying information, the discrete time-varying information obtained is not necessarily related information in the preset time range, but can also be discrete time-varying information in other time ranges.

[0112] In step S520, the discrete time-varying information is diffused into time-varying data at multiple timestamps based on a pre-constructed time-varying information prediction model.

[0113] The present application does not limit the specific time-varying information prediction model or the training process of the time-varying information prediction model. Existing time-varying information prediction models can be used, or a large number of time-varying information samples can be collected to train a time-varying information prediction model.

[0114] The discrete time-varying information obtained is input into the time-varying information prediction model, and the time-varying data of all training data samples at continuous multiple timestamps can be obtained.

[0115] According to some embodiments, the method further comprises steps S150-S170.

[0116] In step S150, a path data set is constructed according to the target 4D space-time field, wherein the path data set includes a plurality of path data samples, and each path data sample includes a starting point sample, an ending point sample, and path information samples from the starting point sample to the ending point sample.

[0117] Based on the constructed target 4D space-time field, end-to-end path planning can also be achieved. Specifically, a plurality of paths (denoted as path information samples) from a point (denoted as a starting point sample) to a point (denoted as an ending point sample) are obtained according to the target 4D space-time field, and a path data set is constructed.

[0118] In step S160, a path planning model is trained according to the path data set.

[0119] The present application does not limit the specific training method of the path planning model. The base model of the path planning model can use any neural network, or the base model of the path planning model can be a model extended based on the target 4D space-time field, for example, adding an embedding head based on the target 4D space-time field as the base model of the path planning model.

[0120] The input of the path planning model is the starting point and the ending point, and the output is the path from the starting point to the ending point.

[0121] In step S170, path planning is performed based on the path planning model.

[0122] The trained model can directly output a feasible path (such as B-spline curve parameters) from the starting point to the ending point according to the input starting point and ending point.

[0123] Further, in some embodiments, further comprising: optimizing the path planning model by reinforcement learning to optimize path smoothness and energy consumption.

[0124] The specific algorithm of reinforcement learning is not limited in the present application, and can be selected according to the situation.

[0125] According to some embodiments, referring to Figure 6 In step S150, the path data set is constructed according to the target 4D space-time field, which can be implemented by steps S1-S4.

[0126] S1: Obtain the occupancy information of a plurality of three-dimensional coordinates according to the target 4D space-time field.

[0127] It can be understood that the target 4D space-time field contains the state information of each three-dimensional coordinate point being occupied or unoccupied, and the present embodiment takes the model output as an example of SDF value and semantic label, but it does not represent the limitation of the present application.

[0128] On this basis, the model inputs three-dimensional coordinates, and obtains the occupancy information of the point according to the SDF value output by the model. Alternatively, the model inputs four-dimensional coordinates and time-varying data (for example, control area coordinates + time window), and obtains the occupancy information of the point according to the SDF value output by the model. It should be noted that

[0129] S2: Select a starting sample and an ending sample from the plurality of three-dimensional coordinates.

[0130] It can be understood that the starting sample and the ending sample can be arbitrarily selected from the plurality of three-dimensional coordinates.

[0131] S3: Determine the path from the starting sample to the ending sample according to the occupancy information, and take the path as the path information sample corresponding to the starting sample and the ending sample, thereby obtaining the path data sample.

[0132] According to the occupancy information of all three-dimensional points, the three-dimensional points that cannot be passed through are avoided, and the path from the starting sample to the ending sample is obtained as the path information sample, thereby constructing the path data sample including the starting sample, the ending sample and the corresponding path information sample.

[0133] S4: Repeat steps S2-S3 to obtain the path data set.

[0134] Repeat the steps to obtain a plurality of path data samples, and the plurality of path data samples constitute the path data set.

[0135] According to some embodiments, after step S160, the method further comprises step S161.

[0136] In step S161, the path planning model is distilled.

[0137] The complex network (for example, 100 MB) is distilled into a lightweight version (<5 MB), and the distilled path planning model has small memory occupation and is suitable for more scenarios, for example, supporting being carried on a UAV on-board computing unit, and assisting path planning of the UAV during task execution.

[0138] According to some embodiments, after step S160, the method further includes step S162.

[0139] In step S162, the path planning model is updated in response to updating of the target 4D spatiotemporal field.

[0140] When the target 4D spatiotemporal field is updated, the path planning model is updated, so as to adjust the accuracy of path planning in real time. The specific updating manner of the path planning model is not limited in the present application, and can be selected according to the situation.

[0141] The device embodiments of the present application are described below, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, reference can be made to the method embodiments of the present application.

[0142] Figure 7 A block diagram of a 4D spatiotemporal field construction device based on neural field reconstruction according to an exemplary embodiment is shown.

[0143] Figure 7 The device shown can execute the aforementioned 4D spatiotemporal field construction method based on neural field reconstruction according to embodiments of the present application.

[0144] As shown in the figure, Figure 7 The 4D spatiotemporal field construction device based on neural field reconstruction can include:

[0145] Referring to Figure 7 and referring to the foregoing description, the data set construction unit 710 is configured to perform supervised information labeling on a plurality of three-dimensional coordinates in the 3D model to be processed, to obtain a plurality of training data samples, and to construct a training data set.

[0146] The neural field construction unit 720 is configured to divide the three-dimensional space into a plurality of cubic grids of the same size, each grid corresponding to a neural network inside, so as to construct a block neural field model.

[0147] The spatiotemporal field training unit 730 is configured to train the block neural field model based on the training data set to obtain a target 4D spatiotemporal field.

[0148] The spatiotemporal field updating unit 740 is configured to update the target 4D spatiotemporal field in response to local data updating of the 3D model to be processed.

[0149] The device performs similar functions to the method provided above, and other functions can be referred to the foregoing description, which will not be repeated here.

[0150] The embodiment of the application discloses an electronic device, comprising: a processor; a memory, storing a computer program, when the computer program is executed by the processor, the processor executes the above instruction generation method.

[0151] For example, referring to the electronic device 800 shown in Figure 8 , Figure 8 The electronic device 800 includes a processor 801 and a memory 803. Wherein, the processor 801 and the memory 803 are connected, such as through the bus 802. Optionally, the electronic device 800 can also include a transceiver 804. It should be noted that the transceiver 804 is not limited to one in actual application, and the structure of the electronic device 800 does not constitute a limitation on the embodiments of the application.

[0152] The processor 801 can be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in the disclosure. The processor 801 can also be a combination of computing functions, such as one or more microprocessor combinations, DSP and microprocessor combinations, etc.

[0153] The bus 802 can include a channel for transmitting information between the above components. The bus 802 can be a PCI (Peripheral Component Interconnect, peripheral component interconnect) bus or an EISA (Extended Industry Standard Architecture, extended industry standard architecture) bus, etc. The bus 802 can be divided into address bus, data bus, control bus, etc. For the convenience of representation, Figure 8 In the above description, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0154] The memory 803 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions; a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions; an EEPROM (Electrically Erasable Programmable Read-Only Memory), a CD-ROM (Compact Disc Read-Only Memory) or other optical disk storage; a magnetic disk storage or other magnetic storage devices; or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this.

[0155] The memory 803 is configured to store application program codes for implementing the solutions of the present application, and the processor 801 is configured to control the execution of the application program codes. The processor 801 is configured to execute the application program codes stored in the memory 803 to implement the content shown in the foregoing method embodiments.

[0156] Figure 8 The electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0157] The embodiments of the present application disclose a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes an instruction generation method.

[0158] It should be understood that, although each step in the flowchart of the accompanying drawings is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0159] The above is only some embodiments of the present application, and it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for constructing a 4D spatiotemporal field based on neural field reconstruction, characterized in that, include: Supervised information annotation is performed on multiple 3D coordinates in the 3D model to be processed to obtain multiple training data samples, thereby constructing a training dataset; The three-dimensional space is divided into multiple cubic grids of the same size, and each grid contains a neural network, thereby constructing a block neural field model. The segmented neural field model is trained based on the training dataset to obtain the target 4D spatiotemporal field. In response to the local data update of the 3D model to be processed, the target 4D spatiotemporal field is updated.

2. The method according to claim 1, characterized in that, The segmented neural field model is trained based on the training dataset to obtain the target 4D spatiotemporal field, including: Obtain the time-varying data and timestamps of the multiple training data samples in the training dataset within a preset time range, and obtain multiple four-dimensional training data samples based on the multiple training data samples and their timestamps; The time-varying data is encoded to obtain a time-varying vector; Using a pre-constructed hybrid loss function, the block neural field model is trained based on the multiple four-dimensional training data samples and their time-varying vectors until a preset termination condition is met, and the training result is output as the target 4D spatiotemporal field.

3. The method according to claim 2, characterized in that, Using a pre-constructed hybrid loss function, the segmented neural field model is trained based on the multiple four-dimensional training data samples and their time-varying vectors until a preset termination condition is met. The training result is then output as the target 4D spatiotemporal field, including: Extract the query vector from the multiple four-dimensional training data samples; Using a preset attention mechanism, the weighted conditional features corresponding to the multiple four-dimensional training data samples are obtained based on the query vector and its corresponding time-varying vector. Using a pre-constructed hybrid loss function, the segmented neural field model is trained based on multiple four-dimensional training data samples and their weighted conditional features until a preset termination condition is met, and the training result is output as the target 4D spatiotemporal field.

4. The method according to claim 1, characterized in that, In response to a local data update of the 3D model to be processed, the target 4D spatiotemporal field is updated, including: In response to the local data update of the 3D model to be processed, one or more neural networks in the target 4D spatiotemporal field corresponding to the updated data are obtained as the target neural network; A local training dataset is constructed based on the updated data, and the target neural network is fine-tuned based on the local training dataset to obtain the updated target 4D spatiotemporal field.

5. The method according to claim 1, characterized in that, Supervised information annotation is performed on multiple 3D coordinates in the 3D model to be processed to obtain multiple training data samples, thereby constructing a training dataset, including: The training dataset is augmented, and the training dataset is updated based on the results of the augmentation.

6. The method according to claim 2 or 3, characterized in that, The hybrid loss function includes: L=L sdf +λL cls ; Where L is the mixture loss function, L sdf Let L be the SDF regression loss function. cls Let λ be the semantic classification loss function, and x be the weight coefficient. i Let y be the coordinates of a point in space. c Here, N represents the true values ​​for the semantic classification categories, and N is the total number of SDF points used for computation. φ is the category estimate for semantic classification, c is the total number of categories in semantic classification, f is the SDF calculation function, and φ is the alias for the network that estimates the SDF.

7. The method according to claim 2, characterized in that, Obtaining the time-varying data and timestamps of the plurality of training data samples in the training dataset within a preset time range includes: Obtain the discrete time-varying information of the plurality of training data samples in the training dataset; Based on a pre-built time-varying information prediction model, the discrete time-varying information is diffused into time-varying data under multiple timestamps.

8. A 4D spatiotemporal field construction device based on neural field reconstruction, characterized in that, include: The dataset construction unit is used to annotate multiple 3D coordinates in the 3D model to be processed with supervised information to obtain multiple training data samples, thereby constructing a training dataset; A neural field construction unit is used to divide a three-dimensional space into multiple cubic grids of the same size, each grid containing a neural network, thereby constructing a block neural field model. The spatiotemporal field training unit is used to train the segmented neural field model based on the training dataset to obtain the target 4D spatiotemporal field. The spatiotemporal field update unit is used to update the target 4D spatiotemporal field in response to the local data update of the 3D model to be processed.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-7.

Images