Method, device and storage medium for dynamically estimating scene structure based on multi-domain spatio-temporal data

Abstract

Embodiments of the present application provide a scene structure dynamic estimation method and device based on multi-domain space-time data and a storage medium, and relate to the technical field of three-dimensional computer vision and three-dimensional surveying and mapping. In the scene structure dynamic estimation scheme based on multi-domain space-time data provided in the present application, a plurality of image sequences are obtained by shooting a target scene from different spatial domains, and then the plurality of image sequences are encoded, the target discrete three-dimensional grid of the scene in each resolution in different domains is estimated from the plurality of image sequences respectively, and alignment and fusion are performed, so that the target discrete three-dimensional grid of the target scene is obtained, and the offset of the target discrete three-dimensional grid at each time is further estimated, so that the three-dimensional structure estimation of the dynamic scene is realized.

Description

Technical Field

[0001] This application relates to the fields of 3D computer vision and 3D mapping technology, and in particular to a method, device and storage medium for dynamic estimation of scene structure based on multi-domain spatiotemporal data. Background Technology

[0002] Scene structure estimation is a technique that obtains the 3D structure of a scene from images. The acquired 3D structural information can support subsequent terrain analysis, 3D mapping, and automatic navigation. Traditional scene structure estimation methods generally only consider image sequences from a single specific domain. When processing multi-domain image sequences, the image sequences of each domain are split into multiple independent domain images, reconstructed separately, and then aligned and fused in 3D space to obtain the final result. Because the alignment process only considers 3D information and ignores information in the 2D image sequence, the results of traditional methods are easily affected by the difference in reconstruction resolution between different domains, resulting in alignment errors. In addition, traditional methods usually only consider the structure estimation of static scenes, and therefore cannot be applied to dynamic scenes.

[0003] In summary, traditional methods are susceptible to the degradation of structure estimation quality due to resolution differences between different domains, and cannot effectively cope with dynamic scenarios. Summary of the Invention

[0004] This application provides a method, device, and storage medium for dynamic estimation of scene structure based on multi-domain spatiotemporal data, which solves the technical problems of existing methods being unable to effectively cope with dynamic scenes and being susceptible to the effects of resolution differences between different domains.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, embodiments of this application provide a method for dynamic estimation of scene structure based on multi-domain spatiotemporal data. The method includes:

[0007] The target scene is captured from N different spatial domains to obtain N sets of image sequences. Each set of image sequences contains at least one image captured from a spatial domain, where N is an integer greater than or equal to 2.

[0008] Encode N sets of image sequences to obtain a three-dimensional tensor F(s,t,c1), where s is a variable indicating the index of the N sets of image sequences, t is a variable indicating the time, and c1 is a variable indicating the image feature channels.

[0009] Based on the three-dimensional tensor F(s,t,c1), target discrete meshes that match the target scene are selected from N discretized three-dimensional tables in N preset spaces. The N discretized three-dimensional tables correspond to N different resolutions, and the target discrete meshes represent the initial estimated shape of the target scene.

[0010] Based on the three-dimensional tensor F(s,t,c1) and the target discrete grid, predict the offset of the target discrete grid at each time indicated by t;

[0011] The target discrete grid and the deviation of the target discrete point at each time step are summed to obtain the target discrete grid after offset at each time step. The target discrete grid after offset at a time step represents the final estimated shape of the target scene at that time step.

[0012] In the above scheme, the resolution differences of image sequences captured in different spatial domains are fully considered in the process of estimating the target discrete grid of the scene. The target discrete grid that matches the target scene is selected from N discrete 3D tables with different resolutions. In contrast, existing methods ignore the resolution differences and only select from discrete 3D tables with the same resolution. Therefore, this scheme can effectively deal with the resolution differences of multi-domain image sequences. At the same time, the discrete grid offset of the dynamic scene at different times is estimated. In contrast, existing methods fix the discrete grid of the scene and treat it as a static scene. Therefore, this scheme can effectively deal with the structure estimation of dynamic scenes.

[0013] In one possible implementation, the method can be applied to a multi-domain spatiotemporal implicit representation model, which includes a multi-domain image coding module, a scene meta-shape representation module, and a scene dynamic deformation field representation module.

[0014] Accordingly, the N sets of image sequences are encoded to obtain a three-dimensional tensor F(s,t,c1), including: inputting the N sets of image sequences into a multi-domain image encoding module, and encoding the N sets of multi-domain image sequences by the multi-domain image encoding module to obtain a three-dimensional tensor F(s,t,c1).

[0015] Accordingly, based on the three-dimensional tensor F(s,t,c1), a target discrete mesh that conforms to the target scene is selected from N discretized three-dimensional tables in N preset spaces. This includes: inputting the three-dimensional tensor F(s,t,c1) and N discretized three-dimensional tables into the scene meta-shape representation module, and having the scene meta-shape representation module select the target discrete mesh that conforms to the target scene from the N discretized three-dimensional tables based on the three-dimensional tensor F(s,t,c1).

[0016] Accordingly, based on the three-dimensional tensor F(s,t,c1) and the target discrete mesh, the offset of the target discrete mesh at each time indicated by t is predicted, including: inputting the three-dimensional tensor F(s,t,c1) and the target discrete mesh into the scene dynamic deformation field representation module, and having the scene dynamic deformation field representation module predict the offset of the target discrete mesh at each time indicated by t based on the three-dimensional tensor F(s,t,c1) and the target discrete mesh.

[0017] In one possible implementation, the scene meta-shape representation module includes N occupancy estimation sub-modules, alignment parameter estimation sub-modules, and multi-domain fusion sub-modules.

[0018] Accordingly, the three-dimensional tensor F(s,t,c1) and N discretized three-dimensional tables are input into the scene meta-shape representation module. The scene meta-shape representation module then selects the target discrete mesh that matches the target scene from the N discretized three-dimensional tables based on the three-dimensional tensor F(s,t,c1), including:

[0019] For each of the N occupancy estimation sub-modules, the following operation is performed: input a three-dimensional tensor F(s,t,c1) and a discretized three-dimensional table into an occupancy estimation sub-module, and then filter out a sub-discrete mesh that matches the target scene.

[0020] The three-dimensional tensor F(s,t,c1) and N sub-discrete meshes are input into the alignment parameter estimation submodule. The alignment parameter estimation submodule estimates the alignment parameter vectors of the sub-discrete meshes at other resolutions relative to the sub-discrete meshes at the lowest resolution. Other resolutions refer to the resolutions other than the lowest resolution among the N resolutions.

[0021] The N sub-discrete meshes and alignment parameter vectors are input into the multi-domain fusion submodule. The multi-domain fusion submodule aligns the sub-discrete meshes at all resolutions to the coordinate system of the sub-discrete mesh at the lowest resolution, thus obtaining the target discrete mesh, which is the discrete mesh obtained by fusing the N sub-discrete meshes.

[0022] In one possible implementation, the alignment parameter vector includes: coordinate rotation parameters of the sub-discrete meshes at other resolutions relative to the sub-discrete meshes at the lowest resolution; and coordinate translation parameters of the sub-discrete meshes at other resolutions relative to the sub-discrete meshes at the lowest resolution.

[0023] In one possible implementation, the multi-domain image encoding module is a two-dimensional convolutional neural network structure. The scene meta-shape representation module is a multilayer perceptron structure containing multiple cascaded fully connected layers. The scene dynamic deformation field representation module is a graph convolutional neural network structure containing multiple cascaded graph convolutional layers.

[0024] In one possible implementation, the N preset spaces have different sizes, each of the N discretized 3D grids in the discretized 3D tables has the same number of grid cells, and one discretized 3D grid corresponds to a set of image sequences captured in a spatial domain.

[0025] In one possible implementation, after obtaining the target discrete grid after offset at each time step, the method may further include:

[0026] The overall loss function L is determined using the following relationship:

[0027] L = L S +L S-T ;

[0028] Among them, L S Let L be the shape loss function. S-T This is the spatiotemporal consistency loss function.

[0029] In one possible implementation, the shape loss function L S The following relationship is used to determine:

[0030]

[0031] Where k1 and k2 are preset proportional coefficients;

[0032] M m Represents the target discrete mesh;

[0033] M* m Represents the theoretical true value of the target discrete mesh;

[0034] M represents the target discrete grid after offset at each time step;

[0035] M* represents the theoretical true value of the target discrete grid after offset at each time step;

[0036] p represents the variable of each independent grid cell in the target discrete grid.

[0037] In one possible implementation, the spatiotemporal consistency loss function L S-T The following relationship is used to determine:

[0038]

[0039] Where ΔM represents the offset of the target discrete grid at each time indicated by t;

[0040] t-1 is the time preceding the time indicated by t;

[0041] t+1 is the next time after the time indicated by t.

[0042] In one possible implementation, the N sets of image sequences include at least: a set of image sequences taken by a satellite visible light imaging device, a set of image sequences taken by a UAV visible light imaging device, and a set of image sequences taken by a ground-based visible light imaging device.

[0043] In a second aspect, a computer device is provided, including a processor and a memory, the processor being coupled to the memory, the processor being configured to execute a computer program or instructions stored in the memory to cause the electronic device to implement the structured light system calibration method as described in any of the first aspects.

[0044] Thirdly, a computer-readable storage medium is provided that stores a computer program, which, when run on a computer device, causes the computer device to perform the structured light system calibration method as described in any of the first aspects.

[0045] Fourthly, a computer program product is provided, which, when run on a computer, causes the computer to execute the structured light system calibration method as described in any of the first aspects.

[0046] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0047] Figure 1 A schematic diagram of a scene structure dynamic estimation system based on multi-domain spatiotemporal data provided in this application embodiment;

[0048] Figure 2 A flowchart illustrating the overall solution provided in the embodiments of this application;

[0049] Figure 3 This is a schematic diagram illustrating the specific structure of the scene element shape representation module provided in the embodiments of this application;

[0050] Figure 4 A flowchart illustrating a scene structure dynamic estimation method based on multi-domain spatiotemporal data provided in this application embodiment;

[0051] Figure 5 A schematic diagram of the module structure of a scene structure dynamic estimation method based on multi-domain spatiotemporal data provided in an embodiment of this application;

[0052] Figure 6 A schematic diagram of a set of dynamic scene structure estimations provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0054] In the description of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. In the description of this application, "and / or" is merely a way of describing the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone.

[0055] The terms “first” and “second” in the specification and claims of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0056] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0057] Traditional scene structure estimation methods typically treat image sequences acquired from multiple spatial domains as image sequences from multiple independent domains and process them separately. This often ignores the spatial resolution differences between different domains, resulting in poor alignment and structure estimation performance. At the same time, traditional methods are often unable to effectively cope with dynamic scenes due to the assumption that they are based on static scenes.

[0058] In view of this technical problem, this application provides a dynamic scene structure estimation scheme based on multi-domain spatiotemporal data: The scheme encodes a sequence of target scene images captured in different spatial domains, estimates discrete 3D meshes of the scene at their respective resolutions in different domains, aligns and fuses these meshes to obtain the discrete 3D meshes of the target scene, and further estimates the offset of the discrete 3D meshes at each time step, thus achieving 3D structure estimation of the dynamic scene. On the one hand, because it fully considers the spatial resolution differences between different domains, and also considers both 3D and 2D image information during the alignment and fusion process, this scheme can effectively address scene structure estimation under multi-domain image sequences. On the other hand, this scheme, while estimating the static structure of the scene, further considers the offset at each time step in the dynamic scene, thus enabling structure estimation of the dynamic scene.

[0059] The following section provides an example illustration of the scene structure dynamic estimation scheme based on multi-domain spatiotemporal data proposed in this application, with reference to the accompanying drawings.

[0060] Figure 1 This is a schematic diagram of a scene structure dynamic estimation system based on multi-domain spatiotemporal data, provided in an embodiment of this application.

[0061] like Figure 1 As shown, the system can include visible light image acquisition devices in multiple spatial domains, such as satellites, drones, and ground-based visible light imaging devices (e.g., ground cameras). The system may also include computer equipment connected to these visible light image acquisition devices via wired or wireless means. The visible light image acquisition devices in each spatial domain are distributed relative to the scene being photographed at different angles, shooting distances, and fields of view.

[0062] In some embodiments, visible light image acquisition devices in multiple space domains, such as satellites, drones, and ground cameras, can be visible light image acquisition devices of any model and parameters. For example, a satellite image acquisition device can be a high-resolution visible light camera with a shooting distance of about 100 km, whose typical ground shooting field of view is about 10 km. 2 The image acquisition equipment for drones can be a high-resolution visible light camera with a shooting distance of about 100m, or multiple visible light cameras that can shoot from different angles simultaneously. Its typical ground shooting field of view is about 100m. 2 The ground-based camera can be a low-resolution visible light camera with a shooting distance of about 10 meters, and its typical field of view is about 10 meters. 2 It should be noted that the fields of view of visible light image acquisition devices in different domains need to overlap to a certain extent.

[0063] Visible light image acquisition devices in multiple space domains, such as satellites, drones, and ground cameras, are used to capture target scenes from different space domains, thereby obtaining N sets of image sequences corresponding to the target scene. For example, the first set of image sequences can be captured by a satellite camera, the second set by a drone, and the third set by a ground camera. At least one image sequence can be captured in each space domain. Each image in each sequence is captured at a different time.

[0064] The computer equipment is used to obtain the discrete grid of the target after each time step of the scene based on N sets of image sequences captured by visible light image acquisition devices in multiple spatial domains such as satellites, drones and ground cameras, thereby realizing the three-dimensional structure estimation of dynamic scenes.

[0065] It should be noted that the above embodiments are illustrated using image sequences acquired by three visible light image acquisition devices: satellite, drone, and ground camera, and do not limit the embodiments of this application. In actual implementation, any two of the three visible light image acquisition devices—satellite, drone, and ground camera—can be used, and of course, visible light image acquisition devices from other spatial domains can also be used.

[0066] Figure 2 The flowchart illustrates the overall solution provided in the embodiments of this application.

[0067] This paper illustrates the application of a scene structure dynamic estimation scheme based on multi-domain spatiotemporal data to a multi-domain spatiotemporal implicit representation model. Figure 2 As shown, the multi-domain spatiotemporal implicit representation model includes a multi-domain image encoding module 01, a scene meta-shape representation module 02, and a scene dynamic deformation field representation module 03. Specifically, the multi-domain image encoding module 01 encodes the input N sets of image sequences to obtain a three-dimensional tensor F(s,t,c1); the scene meta-shape representation module 02 selects the target discrete mesh of the target scene from N discretized three-dimensional tables based on the input three-dimensional tensor F(s,t,c1); and the scene dynamic deformation field representation module 03 estimates the offset of the target discrete mesh at each time step based on the three-dimensional tensor F(s,t,c1) and the target discrete mesh.

[0068] After capturing N image sequences of the target scene using visible light image acquisition devices in multiple spatial domains, such as satellites, drones, and ground cameras, these N image sequences are input into a multi-domain image encoding module 01. The multi-domain image encoding module 01 encodes these N image sequences to obtain a three-dimensional tensor F(s,t,c1), meaning the output of the multi-domain image encoding module 01 is the three-dimensional tensor F(s,t,c1). Then, the three-dimensional tensor F(s,t,c1) is input into both the scene meta-shape representation module 02 and the scene dynamic deformation field representation module 03. The scene meta-shape representation module 02 processes the three-dimensional tensor F(s,t,c1) and the input N discretized three-dimensional tables at different resolutions to obtain an estimated target discrete mesh; meaning the output of the scene meta-shape representation module 02 is the target discrete mesh. The scene dynamic deformation field representation module 03 processes the three-dimensional tensor F(s,t,c1) and the target discrete mesh to estimate the offset of the target discrete mesh at each time step; meaning the output of the scene dynamic deformation field representation module 03 is the offset of the target discrete mesh at each time step. Finally, by combining the target discrete mesh and its offset, dynamic estimation of the scene's 3D structure can be achieved.

[0069] It should be noted that the above embodiments are illustrated using a multi-domain spatiotemporal implicit representation model to achieve dynamic estimation of the three-dimensional structure of a scene, and do not limit the embodiments of this application. In actual implementation, any other possible representation model can also be used to achieve dynamic estimation of the three-dimensional structure of a scene.

[0070] Figure 3 This is a schematic diagram of the specific structure of the scene element shape representation module 02 provided in the embodiments of this application.

[0071] like Figure 3 As shown, the scene element shape representation module 02 includes N occupancy estimation submodules 021, alignment parameter estimation submodules 022, and multi-domain fusion submodules 023. Based on the input 3D tensor F(s,t,c1) and N discretized 3D tables, the occupancy estimation submodule estimates N sub-discrete meshes. Then, the alignment parameter estimation submodule estimates the alignment parameter vectors of the N sub-discrete meshes by analyzing the 3D tensor and the N sub-discrete meshes. Finally, the multi-domain fusion submodule aligns and fuses the N sub-discrete meshes according to the estimated alignment parameter vectors to obtain the target discrete mesh.

[0072] In the dynamic estimation of scene structure based on multi-domain spatiotemporal data, the method provided in this application inputs N different resolutions into the scene metashape representation module, which can fully consider the spatial resolution differences between different domains. At the same time, the scene dynamic deformation representation module can effectively capture the dynamic offset of the scene at different times. Therefore, compared with traditional scene structure estimation methods, the method provided in this application can effectively handle multi-domain image sequences and dynamic scenes, and improve the accuracy of structure estimation.

[0073] Figure 4 This is a flowchart illustrating a method for dynamic estimation of scene structure based on multi-domain spatiotemporal data, provided in an embodiment of this application. The execution entity of this method can be a computer device or a functional module within a computer device; this embodiment does not impose such a limitation. Accordingly, Figure 5 To and Figure 4 The corresponding module structure diagram. For example... Figure 4 As shown, the method may include the following S1 to S5.

[0074] S1. Take pictures of the target scene from N different spatial domains to obtain N sets of image sequences.

[0075] The N image sequences can be acquired by visible light image acquisition devices in multiple space domains, such as satellites, drones, and ground cameras. For example, a satellite captures the target scene from space, obtaining the first image sequence; a drone captures the target scene from the sky, obtaining the second image sequence; and a ground camera or handheld camera captures the target scene from the ground, obtaining the Nth image sequence. Here, N is an integer not less than 2. Each image sequence contains at least one image.

[0076] Each image sequence captured in each space domain can simultaneously contain multiple scene image sub-sequences captured by various types of satellites (or drones, ground cameras, etc.) under different angles, shooting distances, shooting fields of view, and different time conditions. These image sub-sequences constitute the same image sequence.

[0077] S2. The computer equipment encodes N sets of image sequences to obtain a three-dimensional tensor F(s,t,c1).

[0078] Specifically, this can be achieved through the following ①-②.

[0079] ① Input N sets of image sequences into the multi-domain image encoding module 01 in sequence. The multi-domain image encoding module 01 performs multiple convolution and pooling encoding operations on them to obtain three two-dimensional tensors f in sequence. i (t,c1). For example, after encoding the first group of image sequences, we get f1(t,c1); after encoding the second group of image sequences, we get f2(t,c1), and so on, until the Nth group of image sequences is encoded. N (t, c1). Where t is a variable indicating time, and c1 is a variable indicating the image feature channels. Generally, c1 can be a positive integer such as 128, 256, or 512, or other positive integers.

[0080] Optionally, the multi-domain image encoding module 01 can be a conventional two-dimensional convolutional neural network, or a multilayer perceptron neural network model, etc., and the embodiments of this application are not limited thereto.

[0081] ② The multi-domain image coding module 01 concatenates N two-dimensional tensors in the third dimension to obtain a three-dimensional tensor F(s,t,c1). The third dimension is the index corresponding to the N sets of image sequences, that is, s in the three-dimensional tensor F(s,t,c1).

[0082] S3. The computer device selects a target discrete mesh that matches the target scene from N discretized 3D tables in N preset spaces based on the 3D tensor F(s,t,c1). The target discrete mesh represents the initial estimated shape of the target scene, also known as the meta-shape.

[0083] Specifically, this can be achieved through the following ①-③:

[0084] ① Based on the three-dimensional tensor F(s,t,c1), each occupancy estimation submodule 021 in the scene element shape representation module 02 selects N sub-discrete grids of the scene from N discretized three-dimensional tables.

[0085] The input N discretized 3D tables are cubic 3D tables composed of K x K unit cubes of size H x H. Each discretized 3D table has the same K but different H values. H reflects the spatial resolution corresponding to the image sequence in different domains. For example, for the first set of image sequences taken by a satellite, the field of view of the satellite image is approximately 100 km. 2 If the resolution of a satellite image is 10000x10000 pixels, then the spatial resolution of the satellite image sequence is 100 / 10000 km = 100 m. Therefore, H can be set to 100 m. Similarly, for the second set of image sequences taken by a drone, the field of view of the drone image is 100 m. 2 If the image resolution is 3000x4000 pixels, then H can be set to 100 / 4000 = 0.025m. Using the same K and different H values, N discretized three-dimensional tables with different spatial resolutions and ranges can be obtained.

[0086] Optionally, the occupancy estimation submodule 021 can be a multilayer perceptron structure containing multiple cascaded fully connected layers. Based on the input 3D tensor F(s,t,c1), the occupancy estimation submodule estimates the occupancy probability P of each cell cube in the N discretized 3D tables. O Greater than the preset threshold T O The unit cube belongs to the scene, and all that satisfy P O ≥T O The unit cubes constitute the sub-discrete mesh.

[0087] ② Based on the 3D tensor F(s,t,c1) and N sub-discrete meshes, the alignment parameter estimation submodule 022 in the scene element shape representation module 02 estimates the alignment parameter vectors of the sub-discrete meshes at other resolutions relative to the sub-discrete mesh at the lowest resolution. Here, "other resolutions" refers to the resolutions other than the lowest resolution among the N resolutions. For example, if the first sub-discrete mesh corresponding to the first set of images captured by the satellite has the highest spatial resolution among the N sub-discrete meshes, then its coordinate system is selected as the reference coordinate system. The alignment parameter vector then includes the rigid body transformation parameters of the other second to N sub-discrete meshes transformed to the coordinate system of the first sub-discrete mesh, namely, the rotation angles around the X, Y, and Z axes and the translations along the X, Y, and Z axes, respectively. Therefore, for N sub-discrete meshes, the length of the alignment parameter vector output by the alignment parameter estimation submodule is 6(N-1).

[0088] ③ Based on the N sub-discrete meshes and the alignment parameter vector, the multi-domain fusion submodule 023 in the scene element shape representation module 02 performs alignment and fusion on the N sub-discrete meshes to obtain the target discrete mesh of the scene. Taking the alignment and fusion of the 2nd sub-discrete mesh and the 1st discrete mesh as an example, the 2nd sub-discrete mesh is rotated according to the 1st to 3rd parameters in the alignment parameter vector, and then translated according to the 4th to 6th parameters in the alignment parameter vector to obtain the alignment and fusion result of the 1st and 2nd sub-discrete meshes. The alignment and fusion of the Nth sub-discrete mesh and the 1st discrete mesh is carried out in the same way, with coordinate rotation performed according to the 6(N-1)-5 to 3(N-1)-3 parameters in the alignment parameter vector, and coordinate translation performed according to the 6(N-1)-2 to 3(N-1) parameters in the alignment parameter vector to obtain the alignment and fusion result of the Nth sub-discrete mesh and the 1st discrete mesh. After all the second to N-1th sub-discrete grids are aligned with the first sub-discrete grid, the target discrete grid of the scene is obtained.

[0089] S4. The computer device predicts the offset of the target discrete grid at each time indicated by t, based on the three-dimensional tensor F(s,t,c1) and the target discrete grid.

[0090] Existing methods often treat the scene to be estimated as a static scene, and the estimated target discrete mesh is the final result. However, in reality, scenes often exhibit significant dynamic characteristics over time, causing the structure of the target scene to dynamically deform. To address this, this application specifically designs a scene dynamic deformation field representation module to predict the offset of the target discrete mesh relative to the target discrete mesh at different times, thus tackling the dynamic scene estimation task.

[0091] For example, the scene dynamic deformation field representation model 03 can receive inputs such as a three-dimensional tensor F(s,t,c1) and a target discrete mesh, and predict the offset of the scene target discrete mesh at different times. The scene dynamic deformation field representation model can be a graph convolutional neural network structure containing multiple cascaded graph convolutional layers.

[0092] S5. The computer equipment sums the target discrete grid and the offset of the target discrete point at each time step to obtain the target discrete grid after offset at each time step.

[0093] Furthermore, in order to complete the parameter training of all modules, the method can also construct an overall loss function L as shown in the following relation (1).

[0094] L = L S +L S-T (1)

[0095] Among them, L S and L S-T These are the shape loss function and the spatiotemporal consistency loss function, respectively.

[0096] For example, the shape loss function L can be determined using the following relation (2). S :

[0097]

[0098] Where k1 and k2 are preset proportional coefficients;

[0099] M m Represents the target discrete grid;

[0100] M* m This represents the theoretical true value of the target discrete mesh;

[0101] M represents the target discrete grid after the offset at each time point;

[0102] M* represents the theoretical true value of the target discrete grid after the offset at each time step;

[0103] p represents the variable of each independent grid cell in the target discrete grid.

[0104] For example, the spatiotemporal consistency loss function L can be determined using the following relation (3). S-T :

[0105]

[0106] Wherein, ΔM represents the offset of the target discrete grid at each time indicated by t.

[0107] t-1 is the time preceding the time indicated by t;

[0108] t+1 is the next time after the time indicated by t.

[0109] Compared to traditional scene structure estimation, which can only handle structure estimation for static scenes with small differences in spatial resolution, the scene structure dynamic estimation method based on multi-domain spatiotemporal data provided in the above embodiments can fully consider the impact of spatial resolution differences in different spatial domains. Furthermore, due to the design of a targeted scene dynamic deformation field representation module, it can better handle structure estimation for dynamic scenes.

[0110] Figure 6 This is a schematic diagram illustrating dynamic scene structure estimation provided in an embodiment of this application. The scene being filmed is a mountain scene that changes continuously over time.

[0111] Figure 6 (a) in the figure represents the dynamic scene prediction result at time t-1. Figure 6 (b) in the figure represents the dynamic scene prediction result at time t. Figure 6 (c) in the figure represents the dynamic scene prediction result at time t. It can be seen that the scene structure has a significant shift at different times, and the proposed scheme can estimate the dynamic shift of the scene at different times well, thus effectively realizing the structure estimation of dynamic scenes.

[0112] This application also provides a computer device including a processor and a memory, the processor being coupled to the memory, the processor being used to execute computer programs or instructions stored in the memory, so that the computer device implements the methods in the above embodiments.

[0113] This application also provides a computer-readable storage medium storing computer instructions; when the computer-readable storage medium is run on a computer device, it causes the computer device to perform the method described above. The computer instructions can be stored in the computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0114] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0115] This application also provides a chip coupled to a memory. The chip is used to read and execute computer programs or instructions stored in the memory to perform the methods described in the above embodiments. The chip can be a general-purpose processor or a dedicated processor. In some embodiments, the chip includes a processor, a transceiver, and a storage medium. The processor supports the computer device in executing the technical solutions described above. The transceiver receives control from the processor to support the computer device in executing the technical solutions described above.

[0116] It should be noted that the chip can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0117] The computer device, computer-readable storage medium, computer program product, and chip provided in the embodiments of this application are all used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects corresponding to the methods provided above, and will not be repeated here.

[0118] It should be understood that the above description is merely to help those skilled in the art better understand the embodiments of this application, and is not intended to limit the scope of the embodiments of this application. Based on the examples given above, those skilled in the art can obviously make various equivalent modifications or changes. For example, some steps in the various embodiments of the above detection method may be unnecessary, or new steps may be added. Alternatively, any combination of two or more of the above embodiments may be used. Such modifications, changes, or combinations also fall within the scope of the embodiments of this application.

[0119] It should also be understood that the above description of the embodiments of this application focuses on highlighting the differences between the various embodiments. Any similarities or differences not mentioned can be referred to each other. For the sake of brevity, they will not be repeated here.

[0120] It should also be understood that the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0121] It should also be understood that in the embodiments of this application, "pre-setting" or "pre-defining" can be achieved by pre-saving the corresponding code, table or other means that can be used to indicate relevant information in the device (e.g., including electronic devices), and this application does not limit the specific implementation method.

[0122] It should also be understood that the methods, situations, categories, and classifications of embodiments in this application are for the convenience of description only and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined without contradiction.

[0123] It should also be understood that, in the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0124] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for dynamic estimation of scene structure based on multi-domain spatiotemporal data, characterized in that, The method includes: The target scene is captured from N different spatial domains to obtain N sets of image sequences. Each set of N image sequences contains at least one image captured in one spatial domain. The N sets of image sequences are encoded to obtain a three-dimensional tensor F(s,t,c1), where s is a variable indicating the index of the N sets of image sequences, t is a variable indicating the time, and c1 is a variable indicating the image feature channel. Based on the three-dimensional tensor F(s,t,c1), a target discrete mesh that conforms to the target scene is selected from N discretized three-dimensional tables in N preset spaces. The N discretized three-dimensional tables correspond to N different resolutions, and the target discrete mesh represents the initial estimated shape of the target scene. Based on the three-dimensional tensor F(s,t,c1) and the target discrete grid, predict the offset of the target discrete grid at each time indicated by t; The target discrete grid and the offset of the target discrete point at each time are summed to obtain the target discrete grid after offset at each time. The target discrete grid after offset at a time represents the final estimated shape of the target scene at that time. Where N is an integer greater than or equal to 2; After obtaining the target discrete grid after each time-time offset, the method further includes: The overall loss function is determined using the following relationship. L : ； in, L S For shape loss function, L S-T The spatiotemporal consistency loss function; Shape loss function L S The following relationship is used to determine: ； in, k 1 and k 2 represents the preset scaling factor; M m Represents the target discrete grid; M* m This represents the theoretical true value of the target discrete mesh; M This represents the target discrete grid after the offset at each said time. M * indicates the theoretical true value of the target discrete grid after the offset at each said time; p The variable representing each independent grid cell in the target discrete grid; Spatiotemporal consistency loss function L S-T The following relationship is used to determine: ； Where, Δ M This represents the offset of the target discrete grid at each time indicated by t; t-1 is the time preceding the time indicated by t; t+1 is the next time after the time indicated by t.

2. The method according to claim 1, characterized in that, The method is applied to a multi-domain spatiotemporal implicit representation model, which includes a multi-domain image encoding module, a scene meta-shape representation module, and a scene dynamic deformation field representation module; wherein... The step of encoding the N sets of image sequences to obtain the three-dimensional tensor F(s,t,c1) includes: inputting the N sets of image sequences into the multi-domain image encoding module, and having the multi-domain image encoding module encode the N sets of multi-domain image sequences to obtain the three-dimensional tensor F(s,t,c1); The step of selecting a target discrete mesh that matches the target scene from N discretized three-dimensional tables in N preset spaces based on the three-dimensional tensor F(s,t,c1) includes: inputting the three-dimensional tensor F(s,t,c1) and the N discretized three-dimensional tables into the scene meta-shape representation module, and having the scene meta-shape representation module select the target discrete mesh that matches the target scene from the N discretized three-dimensional tables based on the three-dimensional tensor F(s,t,c1); The step of predicting the offset of the target discrete mesh at each time indicated by t based on the three-dimensional tensor F(s,t,c1) and the target discrete mesh includes: inputting the three-dimensional tensor F(s,t,c1) and the target discrete mesh into the scene dynamic deformation field representation module, and having the scene dynamic deformation field representation module predict the offset of the target discrete mesh at each time indicated by t based on the three-dimensional tensor F(s,t,c1) and the target discrete mesh.

3. The method according to claim 2, characterized in that, The scene meta-shape representation module includes N occupancy rate estimation sub-modules, alignment parameter estimation sub-modules, and multi-domain fusion sub-modules; The step of inputting the three-dimensional tensor F(s,t,c1) and the N discretized three-dimensional tables into the scene meta-shape representation module, and having the scene meta-shape representation module filter out the target discrete mesh that matches the target scene from the N discretized three-dimensional tables based on the three-dimensional tensor F(s,t,c1), includes: For each of the N occupancy estimation sub-modules, the following operation is performed: input the three-dimensional tensor F(s,t,c1) and a discretized three-dimensional table into an occupancy estimation sub-module, and the occupancy estimation sub-module filters out a sub-discrete grid that matches the target scene; The three-dimensional tensor F(s,t,c1) and N sub-discrete meshes are input into the alignment parameter estimation submodule. The alignment parameter estimation submodule estimates the alignment parameter vectors of the sub-discrete meshes at other resolutions relative to the sub-discrete meshes at the lowest resolution. The other resolutions refer to the resolutions other than the lowest resolution among the N resolutions. The N sub-discrete meshes and the alignment parameter vector are input into the multi-domain fusion submodule. The multi-domain fusion submodule aligns the sub-discrete meshes at all resolutions to the coordinate system of the sub-discrete mesh at the lowest resolution to obtain the target discrete mesh, which is the discrete mesh obtained by fusing the N sub-discrete meshes.

4. The method according to claim 3, characterized in that, The alignment parameter vector includes: The coordinate rotation parameters of the sub-discrete mesh at the other resolutions relative to the sub-discrete mesh at the lowest resolution; and the coordinate translation parameters of the sub-discrete mesh at the other resolutions relative to the sub-discrete mesh at the lowest resolution.

5. The method according to claim 2, characterized in that, The multi-domain image encoding module is a two-dimensional convolutional neural network structure; The scene element shape representation module is a multilayer perceptron structure containing multiple cascaded fully connected layers; The scene dynamic deformation field representation module is a graph convolutional neural network structure containing multiple cascaded graph convolutional layers.

6. The method according to any one of claims 1 to 5, characterized in that, The N preset spaces have different sizes, each of the N discretized three-dimensional grids in the discretized three-dimensional tables has the same number of grid cells, and one discretized three-dimensional grid corresponds to a set of image sequences captured in a spatial domain.

7. The method according to any one of claims 1 to 5, characterized in that, The N sets of image sequences include at least: A sequence of images captured by a satellite visible light imaging device; A sequence of images captured by a drone's visible light imaging device; And a sequence of images taken by a ground-based visible light imaging device.

8. A computer device, characterized in that, The device includes a processor and a memory, the processor being coupled to the memory, the processor being configured to execute computer programs or instructions stored in the memory, such that the computer device implements the scene structure dynamic estimation method based on multi-domain spatiotemporal data as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer device, causes the computer device to perform the scene structure dynamic estimation method based on multi-domain spatiotemporal data as described in any one of claims 1 to 7.

Images