4D spatio-temporal field construction method and device based on neural field reconstruction

Abstract

The application provides a 4D space-time field construction method and device based on neural field reconstruction, and relates to the technical field of data processing. The 4D space-time field construction method based on neural field reconstruction comprises the following steps: performing supervision information labeling on a plurality of three-dimensional coordinates in a to-be-processed 3D model to obtain a plurality of training data samples, thereby constructing a training data set; dividing a three-dimensional space into a plurality of cubic grids of the same size, each grid corresponding to a neural network inside, thereby constructing a block neural field model; training the block neural field model based on the training data set to obtain a target 4D space-time field; and updating the target 4D space-time field in response to local data update of the to-be-processed 3D model. Through block neural field modeling and a continuous learning mechanism, the application converts dynamic update of the 4D space-time field into an incremental inference problem of the neural network, thereby realizing efficient update of the 4D space-time field.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method and apparatus for constructing a 4D spatiotemporal field based on neural field reconstruction. Background Technology

[0002] Currently, in low-altitude operating systems, the 4D spatiotemporal field needs to simultaneously express three-dimensional spatial geometric information (such as terrain, buildings, and obstacles) and dynamic changes in the time dimension (such as weather updates and control zone adjustments).

[0003] Existing technologies mostly use direct storage of 3D models (such as point clouds and meshes) and rely on dedicated databases to achieve dynamic updates. However, the reliance on explicit 3D models (such as meshes and voxels) makes it difficult to cope with the efficient update requirements of dynamic environments (such as temporary control zones and sudden weather events). Summary of the Invention

[0004] Based on this, this application provides a method and apparatus for constructing a 4D spatiotemporal field based on neural field reconstruction, which solves the problem of high dependence on explicit 3D models in the prior art and realizes the construction of a 4D spatiotemporal field that can be updated efficiently.

[0005] According to one aspect of this application, a method for constructing a 4D spatiotemporal field based on neural field reconstruction is proposed, comprising: supervising the annotation of multiple three-dimensional coordinates in the 3D model to be processed to obtain multiple training data samples, thereby constructing a training dataset; dividing the three-dimensional space into multiple cubic grids of the same size, each grid having a corresponding neural network, thereby constructing a block neural field model; training the block neural field model based on the training dataset to obtain the target 4D spatiotemporal field; and updating the target 4D spatiotemporal field in response to the local data update of the 3D model to be processed.

[0006] According to some embodiments, training a segmented neural field model based on a training dataset to obtain a target 4D spatiotemporal field includes: acquiring time-varying data and timestamps of multiple training data samples within a preset time range from the training dataset, and obtaining multiple four-dimensional training data samples based on the multiple training data samples and their timestamps; encoding the time-varying data to obtain time-varying vectors; and training the segmented neural field model using a pre-constructed hybrid loss function based on the multiple four-dimensional training data samples and their time-varying vectors until a preset termination condition is reached, and outputting the training result as the target 4D spatiotemporal field.

[0007] According to some embodiments, a block neural field model is trained using a pre-constructed hybrid loss function based on multiple four-dimensional training data samples and their time-varying vectors until a preset termination condition is met, and the training result is output as the target 4D spatiotemporal field. This includes: extracting query vectors from multiple four-dimensional training data samples; using a preset attention mechanism, obtaining weighted conditional features corresponding to multiple four-dimensional training data samples based on the query vectors and their corresponding time-varying vectors; and using a pre-constructed hybrid loss function to train the block neural field model based on multiple four-dimensional training data samples and their weighted conditional features until a preset termination condition is met, and outputting the training result as the target 4D spatiotemporal field.

[0008] According to some embodiments, in response to a local data update of the 3D model to be processed, the target 4D spatiotemporal field is updated, including: in response to a local data update of the 3D model to be processed, obtaining one or more neural networks in the target 4D spatiotemporal field corresponding to the updated data as target neural networks; constructing a local training dataset based on the updated data, and fine-tuning the target neural network based on the local training dataset to obtain the updated target 4D spatiotemporal field.

[0009] According to some embodiments, supervised information annotation is performed on multiple three-dimensional coordinates in the 3D model to be processed to obtain multiple training data samples, thereby constructing a training dataset, including: performing data augmentation on the training dataset, and updating the training dataset based on the results of data augmentation.

[0010] According to some embodiments, the hybrid loss function includes:

[0011] L = L sdf +λL cls ;

[0012]

[0013] Where L is the mixture loss function, L sdf Let L be the SDF regression loss function. cls Let λ be the semantic classification loss function, and x be the weight coefficient. i Let y be the coordinates of a point in space. c Here, N represents the true values ​​for the semantic classification categories, and N is the total number of SDF points used for computation. φ is the category estimate for semantic classification, c is the total number of categories in semantic classification, f is the SDF calculation function, and φ is the alias for the network that estimates the SDF.

[0014] According to some embodiments, obtaining time-varying data and their timestamps of multiple training data samples in a training dataset within a preset time range includes: obtaining discrete time-varying information of multiple training data samples in the training dataset; and based on a pre-built time-varying information prediction model, diffusing the discrete time-varying information into time-varying data under multiple timestamps.

[0015] According to some embodiments, the method further includes: constructing a path dataset based on the target 4D spatiotemporal field, wherein the path dataset includes multiple path data samples, the path data samples include a starting point sample, an ending point sample, and path information samples from the starting point sample to the ending point sample; training a path planning model based on the path dataset; and performing path planning based on the path planning model.

[0016] According to some embodiments, a path dataset is constructed based on the target 4D spatiotemporal field, including: S1: Obtaining occupancy information of multiple three-dimensional coordinates based on the target 4D spatiotemporal field; S2: Selecting a start-point sample and an end-point sample from the multiple three-dimensional coordinates; S3: Determining the path from the start-point sample to the end-point sample based on the occupancy information, and using the path as the path information sample corresponding to the start-point sample and the end-point sample, thereby obtaining a path data sample; S4: Repeating steps S2-S3 to obtain the path dataset.

[0017] According to some embodiments, the method further includes distilling the path planning model.

[0018] According to some embodiments, the method further includes updating the path planning model in response to an update of the target 4D spatiotemporal field.

[0019] According to one aspect of this application, a 4D spatiotemporal field construction device based on neural field reconstruction includes: a dataset construction unit for supervising and labeling multiple three-dimensional coordinates in a 3D model to be processed to obtain multiple training data samples, thereby constructing a training dataset; a neural field construction unit for dividing the three-dimensional space into multiple cubic grids of the same size, each grid containing a corresponding neural network, thereby constructing a segmented neural field model; a spatiotemporal field training unit for training the segmented neural field model based on the training dataset to obtain a target 4D spatiotemporal field; and a spatiotemporal field update unit for updating the target 4D spatiotemporal field in response to local data updates in the 3D model to be processed.

[0020] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0021] According to one aspect of this application, a computer-readable medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.

[0022] Through the embodiments provided in this application, supervised annotation is performed on multiple three-dimensional coordinates in the 3D model to be processed to construct a training dataset; the three-dimensional space is divided into multiple cubic grids, such that each grid corresponds to a neural network, and a block neural field model is constructed. The block neural field model is trained based on the training dataset to obtain the target 4D spatiotemporal field. The target 4D spatiotemporal field can be fine-tuned according to the local data update of the 3D model to be processed. This application transforms the dynamic update of the 4D spatiotemporal field into an incremental inference problem of the neural network through block neural field modeling and continuous learning mechanism, thereby achieving efficient update of the 4D spatiotemporal field. Attached Figure Description

[0023] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0025] Figure 1 A flowchart of a 4D spatiotemporal field construction method based on neural field reconstruction provided for embodiments of this application;

[0026] Figure 2 The flowchart provided in this application illustrates the process of training a segmented neural field model based on a training dataset to obtain the target 4D spatiotemporal field.

[0027] Figure 3 The flowchart provided in this application embodiment uses a pre-constructed hybrid loss function to train a block neural field model based on multiple four-dimensional training data samples and their time-varying vectors until a preset termination condition is reached, and outputs the training results as a flowchart of the target 4D spatiotemporal field.

[0028] Figure 4 A flowchart illustrating how the target 4D spatiotemporal field is updated in response to local data updates of the 3D model to be processed, as provided in this embodiment of the application.

[0029] Figure 5 A flowchart for obtaining time-varying data and timestamps of multiple training data samples in a training dataset within a preset time range, provided for embodiments of this application;

[0030] Figure 6 A flowchart for constructing a path dataset based on a target 4D spatiotemporal field, provided in an embodiment of this application;

[0031] Figure 7 A block diagram of a 4D spatiotemporal field construction device based on neural field reconstruction provided in an embodiment of this application;

[0032] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0037] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0038] For specific implementation details, please refer to the following examples.

[0039] Figure 1 A flowchart illustrating a 4D spatiotemporal field construction method based on neural field reconstruction provided in this application embodiment. Figure 1 As shown, the method includes steps S110-S140.

[0040] In step S110, supervised information is labeled on multiple three-dimensional coordinates in the 3D model to be processed to obtain multiple training data samples, thereby constructing a training dataset.

[0041] The 3D model to be processed as referred to in this application includes 3D models of entities (such as terrain, buildings, and obstacles) within a three-dimensional space of the target area. This application does not limit its representation format; the representation format of the 3D model to be processed includes point clouds, meshes, and voxels for computation, and formats such as OSGB, GLB, GLTF, and 3D Tiles for display. Furthermore, scene and asset files commonly used in the game industry (such as OBJ, STL, and FBX) can also be used as representation formats.

[0042] The target region refers to the three-dimensional region related to the 4D spatiotemporal field to be constructed. It should be emphasized that the 4D spatiotemporal field referred to in this application is a continuous spatiotemporal structure that incorporates time as the fourth dimension and forms a three-dimensional space. It includes three-dimensional spatial geometric information and dynamic changes in the time dimension, i.e., a near real-time updated three-dimensional space.

[0043] Multiple 3D coordinates are extracted from the 3D model to be processed. The extracted 3D coordinates are labeled with supervised information to construct multiple training data samples, thereby obtaining the training dataset.

[0044] According to the example embodiment, the supervision information annotation includes at least one of symbolic occupancy field annotation (SDF value annotation) and semantic label annotation.

[0045] In step S120, the three-dimensional space is divided into multiple cubic grids of the same size, and each grid contains a neural network, thereby constructing a block neural field model.

[0046] This application utilizes neural radiation fields to construct 4D spatiotemporal fields, aiming to improve upon traditional 4D spatiotemporal fields that struggle to meet the demands of efficient updates in dynamic environments.

[0047] It is important to explain that neural implicit field technology (such as SDF networks) implicitly encodes geometric information through neural networks, possessing advantages such as continuous representation, dynamic updating, and lightweight design. In other words, neural implicit field technology can continuously represent data without the need for discretized storage, supports queries at any resolution, and can effectively improve the storage efficiency and flexibility of traditional 4D spatiotemporal fields. Neural implicit field technology can be dynamically updated, and local model corrections can be achieved through incremental training, which can meet the high-efficiency update requirements of dynamic environments. A single neural implicit field technology network can be compressed to the MB level, making it suitable for edge deployment.

[0048] Based on this, this application utilizes neural implicit field technology, employing a grid to divide a three-dimensional space, with each grid containing a corresponding neural network. The input to the neural network is three-dimensional coordinates (x, y, z), and the output is the corresponding content labeled with supervisory information.

[0049] According to the example implementation, the supervision information is labeled as SDF value labels and semantic labels (such as "building", "vegetation", etc.), and the corresponding neural network output is SDF value and semantic label.

[0050] This application does not restrict the specific model of the neural network; the appropriate model can be selected based on the actual situation.

[0051] According to the example embodiment, each three-dimensional spatial grid corresponds to one MLP (Multilayer Perceptron).

[0052] Multiple neural networks are superimposed according to their corresponding three-dimensional spatial grids to construct a segmented neural field model.

[0053] In step S130, a segmented neural field model is trained based on the training dataset to obtain the target 4D spatiotemporal field.

[0054] The block neural field model is trained and optimized using the training dataset. This application does not restrict the specific training algorithm, which can be selected according to the actual situation. The training results are output as the target 4D spatiotemporal field.

[0055] In step S140, the target 4D spatiotemporal field is updated in response to the local data update of the 3D model to be processed.

[0056] When local changes occur in the 3D model to be processed, only the network of the corresponding block is fine-tuned.

[0057] According to the example embodiment, a new dataset is constructed based on the updated local data of the local region, and the target 4D spatiotemporal field is fine-tuned using the new dataset.

[0058] Furthermore, in some embodiments, this application further includes: during the update process in step S140, using the Elastic Weight Consolidation (EWC) algorithm to limit the update magnitude of key parameters in order to prevent catastrophic forgetting.

[0059] Based on the above embodiments, this application can achieve a network inference latency of <10ms (single NVIDIA T4 GPU) and good real-time inference capability.

[0060] This application also supports batch queries, which improves throughput.

[0061] This application supervises the annotation of multiple 3D coordinates in the 3D model to be processed to construct a training dataset; divides the 3D space into multiple cubic grids, so that each grid corresponds to a neural network, constructing a block neural field model; trains the block neural field model based on the training dataset to obtain the target 4D spatiotemporal field; the target 4D spatiotemporal field can be fine-tuned according to the local data update of the 3D model to be processed; this application transforms the dynamic update of the 4D spatiotemporal field into an incremental inference problem of neural networks through block neural field modeling and continuous learning mechanism, thereby achieving efficient update of the 4D spatiotemporal field.

[0062] According to some embodiments, refer to Figure 2 In step S130, a segmented neural field model is trained based on the training dataset to obtain the target 4D spatiotemporal field, which can be specifically achieved through steps S210-S230.

[0063] In step S210, time-varying data and timestamps of multiple training data samples in the training dataset within a preset time range are obtained, and multiple four-dimensional training data samples are obtained based on the multiple training data samples and their timestamps.

[0064] Specifically, a timestamp input (t) needs to be added to each block network to model time-varying characteristics (such as weather diffusion).

[0065] In practice, the first step is to acquire time-varying data and their timestamps from multiple training data samples within a preset time range in the training dataset. The preset time range refers to the sampling range, which can be selected based on the actual situation.

[0066] To increase the accuracy of the trained model, a time range covering various time-varying characteristics is selected as the preset time range, and the timestamps corresponding to the time-varying data of the time-varying characteristics are obtained.

[0067] Add timestamps to the training data samples to obtain four-dimensional training data samples.

[0068] In step S220, the time-varying data is encoded to obtain a time-varying vector.

[0069] According to the example implementation, Transformer is used to encode time series data to capture long-term dependencies.

[0070] In step S230, the block neural field model is trained using a pre-constructed hybrid loss function based on multiple four-dimensional training data samples and their time-varying vectors until the preset termination condition is met, and the training result is output as the target 4D spatiotemporal field.

[0071] The segmented neural field model is trained using a pre-built hybrid loss function.

[0072] During training, time-varying vectors can be embedded into four-dimensional training data samples in the form of annotations, or they can be fused with four-dimensional training data samples through an attention mechanism to participate in training.

[0073] The preset termination condition may include the number of iterations or the condition that the preset evaluation index is met; this application does not impose any restrictions on this.

[0074] If the preset termination conditions are met, the training results are output as the target 4D spatiotemporal field.

[0075] According to some embodiments, refer to Figure 3 In step S230, the block neural field model is trained using a pre-constructed hybrid loss function based on multiple four-dimensional training data samples and their time-varying vectors until the preset termination condition is met, and the training result is output as the target 4D spatiotemporal field. This can be achieved through steps S310-S330.

[0076] In step S310, a query vector is extracted from multiple four-dimensional training data samples.

[0077] Extract the backbone input (x,y,z,t) as the query vector.

[0078] Where (x,y,z) are three-dimensional coordinates and t is a timestamp.

[0079] In step S320, a preset attention mechanism is used to obtain weighted conditional features corresponding to multiple four-dimensional training data samples based on the query vector and its corresponding time-varying vector.

[0080] This embodiment employs a multi-condition fusion strategy based on an attention mechanism to integrate external time-varying conditions such as weather and temporary control zones. The core idea is to allow the neural network to dynamically monitor and weight the influence of different external conditions based on the current time and spatial location, thereby adjusting the network's output. The attention mechanism determines which conditions are more important at the present time and place, assigning them higher weights.

[0081] It should be noted that external condition inputs (such as meteorological data, control zone coordinates + time window, etc.) are time-varying data, which are pre-encoded into time-varying vector format and then enter the attention mechanism-related modules. For example, according to the example implementation, meteorological data (temperature, wind speed, humidity) are concatenated into a time-varying vector v_weather; control zone information (latitude and longitude range, effective time, and expiration time) is encoded into a time-varying vector v_zone; other time-varying data such as traffic and airspace restrictions can also be encoded into corresponding time-varying vectors as appropriate.

[0082] These external features are uniformly mapped through a pre-defined attention mechanism in conjunction with a neural network to obtain weighted conditional features.

[0083] According to the example implementation, a small MLP is used for feature mapping to form keys and values, and the standard Scaled Dot-Product Attention mechanism is selected as the preset attention mechanism:

[0084]

[0085] Where Q is the query vector (extracted from (x,y,z,t)); K is the key vector (the representation after encoding external conditions, i.e., the variable vector); V is the value vector (corresponding to the feature influence strength of each condition); and d k This is a scaling factor to prevent the inner product from becoming too large, which could lead to gradient vanishing.

[0086] Based on this, attention weights are applied to the value vector to obtain the weighted fusion conditional features, which are called weighted conditional features.

[0087] It should be emphasized that the preset attention mechanism can be a multi-head attention mechanism, or a self-attention and single-head attention mechanism, and this application does not limit it.

[0088] In step S330, the block neural field model is trained using a pre-constructed hybrid loss function based on multiple four-dimensional training data samples and their weighted conditional features until the preset termination condition is met, and the training result is output as the target 4D spatiotemporal field.

[0089] The weighted conditional features are concatenated or added to the output of the intermediate layers of the backbone network, and subsequent network layers will use this fused feature to predict the final model output.

[0090] For example, in one embodiment, when a spatiotemporal point falls within a time window of a temporary air traffic control zone, the attention mechanism assigns a higher weight to that region. The model automatically increases the SDF output value of that region (i.e., makes the surface "harder"), thus treating it as an impassable region in path planning.

[0091] According to some embodiments, refer to Figure 4 In step S140, in response to the local data update of the 3D model to be processed, the target 4D spatiotemporal field is updated, which can be specifically implemented through steps S410-S420.

[0092] In step S410, in response to the local data update of the 3D model to be processed, one or more neural networks in the target 4D spatiotemporal field corresponding to the updated data are obtained as the target neural network.

[0093] After the 4D spatiotemporal field is constructed, when data updates are detected in a local area (e.g., a new building is added), the neural network related to the updated local area is located. Specifically, the grid related to the local area in 3D space is located and denoted as the target grid, and then one or more neural networks related to the target grid are determined as the target neural network.

[0094] In step S420, a local training dataset is constructed based on the updated data, and the target neural network is fine-tuned based on the local training dataset to obtain the updated target 4D spatiotemporal field.

[0095] In the actual update process, a continuous learning mechanism is used to construct a new local training dataset using the updated data of the updated region to fine-tune the target neural network.

[0096] This application only updates the neural network corresponding to the affected region to obtain the updated 4D spatiotemporal field, which is time-saving and low-complexity.

[0097] According to some embodiments, in step S110, multiple three-dimensional coordinates in the 3D model to be processed are labeled with supervised information to obtain multiple training data samples, thereby constructing a training dataset, and step S111 is also included.

[0098] In step S111, data augmentation is performed on the training dataset, and the training dataset is updated based on the results of the data augmentation.

[0099] Based on the training dataset constructed from multiple training data samples, data augmentation can be performed on the training data samples to increase the number of training samples and improve the robustness of the model.

[0100] According to an example embodiment, data augmentation specifically includes adding noise (e.g., ±5% coordinate perturbation) to training data samples.

[0101] Furthermore, in some embodiments, based on the supervision annotations of SDF value annotations and semantic label annotations, the hybrid loss function is constructed by combining SDF regression loss (L1 Loss) and semantic classification loss (Cross-Entropy). In this case, the model output is SDF value and / or semantic label.

[0102] Based on the above embodiments, the hybrid loss function includes:

[0103] L = L sdf +λL cls ;

[0104]

[0105] Where L is the mixture loss function, L sdf Let L be the SDF regression loss function. cls Let λ be the semantic classification loss function, and x be the weight coefficient. i Let y be the coordinates of a point in space. c Here, N represents the true values ​​for the semantic classification categories, and N is the total number of SDF points used for computation. Here, is the category estimate for semantic classification, c is the total number of categories in semantic classification, f is the SDF calculation function, and φ is the alias for the network that estimates the SDF. It should be noted that the categories in semantic classification are the same as the semantic labels.

[0106] According to some embodiments, refer to Figure 5 In step S210, time-varying data and their timestamps of multiple training data samples in the training dataset within a preset time range are obtained, which can be achieved through steps S510-S520.

[0107] In step S510, discrete time-varying information of multiple training data samples in the training dataset is obtained.

[0108] It should be noted that some time-varying data may be discrete, such as meteorological data at the minute or hour level.

[0109] Based on this, discrete time-varying information needs to be used for prediction and filling to obtain continuous time-varying information.

[0110] The first step is to acquire discrete time-varying information.

[0111] It should be emphasized that the purpose of step S210 is to obtain time-varying data within a preset time range. However, since this embodiment predicts continuous time-varying data based on discrete time-varying information, the obtained discrete time-varying information may not necessarily be related information within the preset time range, but may also be discrete time-varying information within other time ranges.

[0112] In step S520, based on the pre-built time-varying information prediction model, the discrete time-varying information is diffused into time-varying data under multiple timestamps.

[0113] This application does not restrict the specific time-varying information prediction model or the training process of the time-varying information prediction model. Existing time-varying information prediction models can be used, or a large number of time-varying information samples can be collected to train the time-varying information prediction model.

[0114] By inputting the acquired discrete time-varying information into the time-varying information prediction model, the time-varying data of all training data samples at multiple consecutive time stamps can be obtained.

[0115] According to some embodiments, the method further includes steps S150-S170.

[0116] In step S150, a path dataset is constructed based on the target 4D spatiotemporal field. The path dataset includes multiple path data samples, including a starting point sample, an ending point sample, and path information samples from the starting point sample to the ending point sample.

[0117] Based on the constructed target 4D spatiotemporal field, end-to-end path planning can also be achieved. Specifically, multiple paths (denoted as path information samples) from points (referred to as starting point samples) to points (referred to as ending point samples) are obtained from the target 4D spatiotemporal field, and a path dataset is constructed.

[0118] In step S160, a path planning model is trained based on the path dataset.

[0119] This application does not restrict the specific training method of the path planning model. The base model of the path planning model can be any neural network. The base model of the path planning model can also be an extended model based on the target 4D spatiotemporal field. For example, an embedding head can be added to the target 4D spatiotemporal field as the base model of the path planning model.

[0120] The path planning model takes a starting point and an ending point as input and outputs the path from the starting point to the ending point.

[0121] In step S170, path planning is performed based on the path planning model.

[0122] The trained model can directly output feasible paths (e.g., B-spline curve parameters) from the starting point to the ending point based on the input starting and ending points.

[0123] Furthermore, in some embodiments, the method further includes optimizing the path planning model through reinforcement learning to improve path smoothness and energy consumption.

[0124] This application does not impose restrictions on the specific algorithms used in reinforcement learning; the choice can be made as appropriate.

[0125] According to some embodiments, refer to Figure 6 In step S150, a path dataset is constructed based on the target 4D spatiotemporal field, which can be specifically implemented through steps S1-S4.

[0126] S1: Obtain the occupancy information of multiple three-dimensional coordinates based on the target's 4D spatiotemporal field.

[0127] It is understood that the target 4D spatiotemporal field contains information about whether each three-dimensional coordinate point is occupied or unoccupied. This embodiment uses the model output as SDF value and semantic label as an example for illustration, but it does not represent a limitation of this application.

[0128] Based on this, the model inputs three-dimensional coordinates, and the occupancy information of the point is obtained based on the SDF value output by the model. Alternatively, the model inputs four-dimensional coordinates and time-varying data (e.g., controlled area coordinates + time window), and the occupancy information of the point is obtained based on the SDF value output by the model. It is important to note that...

[0129] S2: Select the start point sample and the end point sample from multiple three-dimensional coordinates.

[0130] Understandably, the starting and ending samples can be arbitrarily selected from multiple three-dimensional coordinates.

[0131] S3: Based on the occupancy information, determine the path from the starting sample to the ending sample, and use the path as the path information sample corresponding to the starting sample and the ending sample to obtain the path data sample.

[0132] Based on the occupancy information of all 3D points, by avoiding 3D points that cannot be passed, the path from the starting point sample to the ending point sample can be obtained as a path information sample, thereby constructing a path data sample including the starting point sample, the ending point sample, and the corresponding path information sample.

[0133] S4: Repeat steps S2-S3 to obtain the path dataset.

[0134] Repeat the steps to obtain multiple path data samples, which together form a path dataset.

[0135] According to some embodiments, after step S160, the method further includes step S161.

[0136] In step S161, the path planning model is distilled.

[0137] Distilling complex networks (e.g., 100MB) into lightweight versions (<5MB) results in a path planning model with a smaller memory footprint, making it suitable for more scenarios. For example, it can be mounted on the onboard computing unit of a drone to assist in path planning during the drone's mission execution.

[0138] According to some embodiments, after step S160, the method further includes step S162.

[0139] In step S162, the path planning model is updated in response to the update of the target 4D spatiotemporal field.

[0140] When the target's 4D spatiotemporal field is updated, the path planning model is updated accordingly, thereby adjusting the accuracy of path planning in real time. This application does not restrict the specific update method of the path planning model; it can be selected as needed.

[0141] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0142] Figure 7 A block diagram of a 4D spatiotemporal field construction apparatus based on neural field reconstruction according to an exemplary embodiment is shown.

[0143] Figure 7 The device shown can perform the aforementioned 4D spatiotemporal field construction method based on neural field reconstruction according to the embodiments of this application.

[0144] like Figure 7 As shown, a 4D spatiotemporal field construction device based on neural field reconstruction may include:

[0145] See Figure 7 Referring to the preceding description, the dataset construction unit 710 is used to annotate multiple three-dimensional coordinates in the 3D model to be processed with supervised information to obtain multiple training data samples, thereby constructing a training dataset.

[0146] The neural field building unit 720 is used to divide the three-dimensional space into multiple cubic grids of the same size, with each grid containing a corresponding neural network, thereby constructing a block neural field model.

[0147] The spatiotemporal field training unit 730 is used to train a segmented neural field model based on the training dataset to obtain the target 4D spatiotemporal field.

[0148] The spatiotemporal field update unit 740 is used to update the target 4D spatiotemporal field in response to the local data update of the 3D model to be processed.

[0149] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0150] This application discloses an electronic device, including: a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to execute the above-described instruction generation method.

[0151] For example, refer to Figure 8 , Figure 8 The illustrated electronic device 800 includes a processor 801 and a memory 803. The processor 801 and the memory 803 are connected, for example, via a bus 802. Optionally, the electronic device 800 may further include a transceiver 804. It should be noted that in practical applications, the transceiver 804 is not limited to one type, and the structure of this electronic device 800 does not constitute a limitation on the embodiments of the present invention.

[0152] Processor 801 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in this disclosure. Processor 801 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0153] Bus 802 may include a pathway for transmitting information between the aforementioned components. Bus 802 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 802 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0154] The memory 803 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other storage medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0155] The memory 803 stores application code that executes the present invention, and its execution is controlled by the processor 801. The processor 801 executes the application code stored in the memory 803 to implement the content shown in the foregoing method embodiments.

[0156] Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0157] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to execute an instruction generation method.

[0158] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0159] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for constructing a 4D spatiotemporal field based on neural field reconstruction, characterized in that, include: Supervised information annotation is performed on multiple 3D coordinates in the 3D model to be processed to obtain multiple training data samples, thereby constructing a training dataset; The three-dimensional space is divided into multiple cubic grids of the same size, and each grid contains a neural network, thereby constructing a block neural field model. The segmented neural field model is trained based on the training dataset to obtain the target 4D spatiotemporal field. In response to the local data update of the 3D model to be processed, the target 4D spatiotemporal field is updated; The segmented neural field model is trained based on the training dataset to obtain the target 4D spatiotemporal field, including: Obtain the time-varying data and timestamps of the multiple training data samples in the training dataset within a preset time range, and obtain multiple four-dimensional training data samples based on the multiple training data samples and their timestamps; The time-varying data is encoded to obtain a time-varying vector; Using a pre-constructed hybrid loss function, the block neural field model is trained based on the multiple four-dimensional training data samples and their time-varying vectors until a preset termination condition is met, and the training result is output as the target 4D spatiotemporal field.

2. The method according to claim 1, characterized in that, Using a pre-constructed hybrid loss function, the segmented neural field model is trained based on the multiple four-dimensional training data samples and their time-varying vectors until a preset termination condition is met. The training result is then output as the target 4D spatiotemporal field, including: Extract the query vector from the multiple four-dimensional training data samples; Using a preset attention mechanism, the weighted conditional features corresponding to the multiple four-dimensional training data samples are obtained based on the query vector and its corresponding time-varying vector. Using a pre-constructed hybrid loss function, the segmented neural field model is trained based on multiple four-dimensional training data samples and their weighted conditional features until a preset termination condition is met, and the training result is output as the target 4D spatiotemporal field.

3. The method according to claim 1, characterized in that, In response to a local data update of the 3D model to be processed, the target 4D spatiotemporal field is updated, including: In response to the local data update of the 3D model to be processed, one or more neural networks in the target 4D spatiotemporal field corresponding to the updated data are obtained as the target neural network; A local training dataset is constructed based on the updated data, and the target neural network is fine-tuned based on the local training dataset to obtain the updated target 4D spatiotemporal field.

4. The method according to claim 1, characterized in that, Supervised information annotation is performed on multiple 3D coordinates in the 3D model to be processed to obtain multiple training data samples, thereby constructing a training dataset, including: The training dataset is augmented, and the training dataset is updated based on the results of the augmentation.

5. The method according to claim 1 or 2, characterized in that, The hybrid loss function includes: ； ； ； in, For a mixed loss function, The SDF regression loss function is... Let λ be the semantic classification loss function, and λ be the weight coefficient. The coordinates of a point in space. For semantic classification, the category truth value, To calculate the total number of SDF points, For semantic classification, the category estimate. c The total number of categories in semantic classification. f For SDF computation functions, The alias for the network used to estimate SDF.

6. The method according to claim 1, characterized in that, Obtaining the time-varying data and timestamps of the plurality of training data samples in the training dataset within a preset time range includes: Obtain the discrete time-varying information of the plurality of training data samples in the training dataset; Based on a pre-built time-varying information prediction model, the discrete time-varying information is diffused into time-varying data under multiple timestamps.

7. The method according to claim 1, characterized in that, The method further includes: A path dataset is constructed based on the target 4D spatiotemporal field, wherein the path dataset includes multiple path data samples, and the path data samples include a starting point sample, an ending point sample, and path information samples from the starting point sample to the ending point sample; A path planning model is trained based on the path dataset. Path planning is performed based on the aforementioned path planning model.

8. The method according to claim 7, characterized in that, A path dataset is constructed based on the target 4D spatiotemporal field, including: S1: Based on the target 4D spatiotemporal field, obtain the occupancy information of the multiple three-dimensional coordinates; S2: Select the start point sample and the end point sample from the plurality of three-dimensional coordinates; S3: Based on the occupancy information, determine the path from the starting sample to the ending sample, and use the path as the path information sample corresponding to the starting sample and the ending sample to obtain the path data sample; S4: Repeat steps S2-S3 to obtain the path dataset.

9. The method according to claim 7, characterized in that, The method further includes: The path planning model is distilled.

10. The method according to claim 7, characterized in that, The method further includes: The path planning model is updated in response to the update of the target 4D spatiotemporal field.

11. A 4D spatiotemporal field construction device based on neural field reconstruction, characterized in that, include: The dataset construction unit is used to annotate multiple 3D coordinates in the 3D model to be processed with supervised information to obtain multiple training data samples, thereby constructing a training dataset; A neural field construction unit is used to divide a three-dimensional space into multiple cubic grids of the same size, each grid containing a neural network, thereby constructing a block neural field model. The spatiotemporal field training unit is used to train the segmented neural field model based on the training dataset to obtain the target 4D spatiotemporal field. The spatiotemporal field update unit is used to update the target 4D spatiotemporal field in response to the local data update of the 3D model to be processed. The spatiotemporal field training unit is also used for: Obtain the time-varying data and timestamps of the multiple training data samples in the training dataset within a preset time range, and obtain multiple four-dimensional training data samples based on the multiple training data samples and their timestamps; The time-varying data is encoded to obtain a time-varying vector; Using a pre-constructed hybrid loss function, the block neural field model is trained based on the multiple four-dimensional training data samples and their time-varying vectors until a preset termination condition is met, and the training result is output as the target 4D spatiotemporal field.

12. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-10.

Images