Intelligent agent coverage planning method based on event flow depth prediction

Abstract

The invention belongs to the technical field of path planning. The invention provides an agent coverage planning method based on event flow depth prediction. According to the embodiment of the invention, the three-dimensional structure information of the environment is constructed online without depending on a point cloud or a structure model known in advance. The method can be combined with a method based on an RGB camera and a depth sensor, and the coverage planning effect is improved. In addition, a required three-dimensional model and an initial observation path point can be provided for a preset class method in advance, so that a shortest coverage path can be searched by further utilizing a TSP solver.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of path planning, in particular to an agent coverage planning method based on event stream deep prediction. BACKGROUND

[0002] With the further development of space and robot technology in China, robot technology will deeply empower the future space field. Currently, some robot technologies have been applied to the aerospace field, such as using a mechanical arm to realize extravehicular maintenance. These technologies have greatly improved the efficiency and safety of extravehicular operations. At present, extravehicular detection and maintenance are mainly realized by manual teleoperation, which is limited by human operation precision, observation angle, bandwidth and time delay. The operation task is relatively simple, and the operation range is relatively limited. Coverage planning technology can further improve the autonomy, efficiency and safety of extravehicular detection and maintenance tasks. Existing coverage planning methods mainly fall into two categories: one is to calculate the coverage path offline and preset in advance, and the other is to predict the coverage path according to the actual observation of the sensor carried by the agent. The coverage planning method preset in advance needs to know the accurate three-dimensional model of the scene in advance, and cannot cope with unknown environments and changes in the scene. The online prediction coverage planning method can make up for the shortcomings of the first method, establish a fine model of the scene, and provide an initial coverage path reference. The coverage planning method preset in advance generates all coverage path points in advance, then models the coverage problem as a TSP problem, and uses a heuristic algorithm to approximately solve it. This approach mainly considers the shortest length of the coverage path. The online prediction coverage planning method needs to generate coverage path points step by step according to existing observations, and select the path with the maximum coverage gain, because it assumes an unknown environment. Because the online prediction-based method has higher applicability and can provide useful information for the first method, it has been widely studied in recent years.

[0003] In the article "Localization, Planning, and Control of a UAV for Rapid Complete Coverage Bridge Inspection in Large-Scale Intermittent GPS Environments", a heuristic method is used to generate a coverage path for the bottom of the bridge, and then a nonlinear optimization-based method is used to calculate the coverage trajectory, and detection experiments are carried out in a real bridge scene. Although this method does not require accurate three-dimensional structure information to plan a time-optimal coverage trajectory, it is still unable to cope with the three-dimensional structure of complex bridges for the main coverage detection task of the bridge bottom.

[0004] In the paper "Near-Optimal 3-D Visual Coverage for Quadrotor Unmanned Aerial Vehicles Under Photogrammetric Constraints", the observation angle constraints of the detection camera and other information are considered to generate coverage observation points, and then a TSP problem solver is used to solve a shortest coverage path that traverses all the observation points. Finally, a continuous coverage trajectory is generated according to the coverage path using a Bezier curve. The method needs to realize the three-dimensional structure information of the known environment, and when the number of coverage observation points is large, the solving efficiency of the TSP problem is low.

[0005] In the paper "Cooperative Motion Planning for Persistent 3D Visual Coverage With Multiple Quadrotor UAVs", the TSP problem is solved using reinforcement learning, which improves the efficiency of solving the TSP problem in the previous method, and extends the above method to multiple UAV coverage. Although the efficiency of this method has been improved, and multiple machines are supported, it still needs accurate three-dimensional information of the known environment. In addition, this method is mainly for three-dimensional terrain coverage tasks, only needs to consider the curved surface structure, and the bridge has a non-convex closed curved surface, so the task is more difficult, and therefore this method may not be suitable.

[0006] In the paper "FC-Planner: A Skeleton-guided Planning Framework for Fast Aerial Coverage of Complex 3D Scenes", the three-dimensional point cloud information of the detected structure is used to divide the space into simple subspaces. First, a coverage order is generated between the subspaces, and then coverage observation points are generated for the point cloud information in each simplified subspace. The path planning problem is modeled as a smaller TSP problem for parallel solving, and finally a simple kinematic trajectory is generated according to the coverage path. Although this method can realize online coverage planning, it needs to realize accurate three-dimensional point cloud information of the known structure, and does not consider the accurate dynamics model of the UAV Therefore, it is necessary to improve one or more problems existing in the above related technical solutions.

[0007] It should be noted that this part aims to provide background or context for the technical solutions of the disclosure stated in the claims. The description herein is not admitted to be prior art because it is included in this part. SUMMARY

[0008] Embodiments of the present disclosure aim to provide an agent coverage planning method based on event stream depth prediction, thereby at least partially overcoming one or more problems caused by limitations and defects of related technologies.

[0009] According to embodiments of the present disclosure, an agent coverage planning method based on event stream depth prediction is provided, comprising: constructing a three-dimensional scene model in a virtual simulation environment, randomly sampling a plurality of camera observation points, generating RGB images, ground truth depth information and camera observation poses, and taking the ground truth depth information as a standard supervision signal; generating event stream data between two adjacent RGB images according to each other; constructing a coverage planning network model, wherein the coverage planning network model comprises an event stream depth prediction network, a voxel occupancy prediction network and a coverage gain prediction network; processing the event stream data using the event stream depth prediction network to output a predicted depth map, taking the depth map at the latest sampling time as a first supervision signal, and generating a point cloud set according to the depth map; processing the point cloud set and the query point using the voxel occupancy prediction network to output voxel occupancy information, and generating a second supervision signal according to the voxel occupancy information at the query point; processing the voxel occupancy information, the current camera observation pose and the historical camera observation pose using the coverage gain prediction network to output the coverage gain of the current camera observation point, and generating a third supervision signal; sampling M new camera points to be moved near the current camera point, and selecting the point with the maximum coverage gain as the next observation point; determining whether the coverage planning network model converges according to the standard supervision signal, the first supervision signal, the second supervision signal and the third supervision signal, and if it converges, a trained coverage planning network model is obtained; wherein the standard supervision signal is used for a supervised learning method to make the event stream depth prediction network converge; the first supervision signal is used for training the depth prediction neural network module; the second supervision signal is used for training the voxel occupancy prediction network; and the third supervision signal is used for training the coverage gain prediction network; applying the trained coverage planning network model to an agent, processing the captured event stream, and selecting the point with the maximum coverage gain as the next observation point until the scene coverage task is completed or the termination condition is reached.

[0010] Further, in the step of constructing a three-dimensional scene model in a virtual simulation environment, randomly sampling a plurality of camera observation points, generating RGB images, ground truth depth information and camera observation poses, and taking the ground truth depth information as a standard supervision signal, comprises: Construct a virtual simulation scene, and import a three-dimensional model of the virtual simulation scene, and set camera parameters; Randomly generate a plurality of camera observation points around the space of the surface of the three-dimensional model, and record the camera observation poses; Capture the corresponding RGB image and ground truth depth information at each camera observation point; The ground truth depth information is used as a standard supervision signal.

[0011] Further, in the step of generating event stream data between two shooting points according to each two adjacent RGB images, comprising: For any two adjacent camera observation points of RGB images, set the agent to move from one point to another at a constant speed in a straight line; Control the agent in the simulation to execute the motion trajectory, and continuously acquire a video sequence; Import the video sequence into the event camera simulator to generate simulated event stream data between the two shooting points.

[0012] Further, the event stream depth prediction network is used to process the event stream data to output a predicted depth map, and the depth map at the latest sampling time is taken as the first supervision signal, and in the step of generating a point cloud set according to the depth map, comprising: Set the depth map of the event stream data to be the depth map at the latest sampling time of the two images, and take it as the first supervision signal of the event stream depth prediction network; Set the historical point cloud information as The current point cloud information converted from the depth map predicted by the event stream depth prediction network is ; The historical point cloud information and the current point cloud information are combined into a point cloud set ; The point cloud set formed by the combination of the two point cloud information is , then

[0013] , wherein is an event frame formed by the event stream data set.

[0014] Further, the voxel occupancy prediction network is used to process the point cloud set and the query point to output the voxel occupancy information, and in the step of generating the second supervision signal according to the voxel occupancy information at the query point, comprising: Four times down-sampling is performed on the point cloud set to obtain point cloud sets of different fineness; Use a self-attention module Collect the K nearest neighbor point clouds of the current point location p in the point cloud set at each level of detail. Encoding nearest neighbor point cloud The features are used to encode and map voxel occupancy information using a multilayer perceptron. :

[0015] in, Let p be the set of K nearest point clouds. This aggregates the point cloud encoding results from four self-attention modules at four different sampling scales. This is the sparse global encoding for the entire current point cloud. Based on the ground truth point cloud information observed under the current camera pose, a second monitoring signal is generated to determine whether the current point is occupied.

[0016] Furthermore, the step of using a coverage gain prediction network to process voxel occupancy information, current camera observation pose, and historical camera observation pose to output the coverage gain of the current camera observation point and generate a third supervision signal includes: The voxel occupancy information was inversely sampled to obtain a set of N discrete sampling points. ; The relationship between sampling points and historical camera observation poses is encoded as spherical harmonic information. ; Use Transformer to aggregate all spherical harmonic information To predict the coverage gain at the current camera observation point:

[0017] in, To obtain K points from the current perspective by performing an inverse sampling operation using occupancy information, This is an inverse sampling operation. The next perspective for network prediction Down Point coverage gain, For the next camera observation pose, To aggregate all Coverage gain at this viewpoint after the point The sum of occupancy rates For point Inverse depth from the current perspective; Using the ground truth depth information of the simulation environment, the true coverage gain of each candidate point is calculated to generate a third supervision signal.

[0018] Further, in the step of sampling M new camera positions to be moved near the current camera position, selecting the position with the maximum coverage gain as the next observation position, comprising: sampling M new camera positions to be moved near the current camera position, and then selecting the position with the maximum coverage gain rate:

[0019] moving the agent to the position.

[0020] Further, in the step of judging whether the coverage planning network model converges according to the standard supervision signal, the first supervision signal, the second supervision signal and the third supervision signal, if converges, obtaining the trained coverage planning network model, comprising: calculating the prediction values of the event flow depth prediction network, the voxel occupancy prediction network and the coverage benefit prediction network according to the standard supervision signal, the first supervision signal, the second supervision signal and the third supervision signal; using the L2 norm result of the difference between the prediction values of the event flow depth prediction network, the voxel occupancy prediction network and the coverage benefit prediction network and the supervision values and the change rate of the L2 norm to judge whether to converge, when the change rate of the L2 norm is less than the first preset threshold, the module converges; or when the L2 norm itself is less than the second preset threshold, the module converges; when the event flow depth prediction network, the voxel occupancy prediction network and the coverage benefit prediction network all converge, the coverage planning network model converges.

[0021] Further, in the step of applying the trained coverage planning network model to the agent, processing the captured event flow, and selecting the position with the maximum coverage gain as the next observation position until completing the scene coverage task or reaching the termination condition, comprising: the agent moves from the current position to the planned next target position at a fixed speed; In this process, the event camera continues to work and collects the event flow in the entire motion process; input the collected event flow into the trained coverage planning network model to obtain the local depth map of the current end position, and convert it into a local point cloud; fuse the local point cloud with the historical point cloud, and update the global implicit occupancy map using the voxel occupancy prediction network; the coverage benefit prediction network receives the current occupancy map and all historical poses, and respectively calculates the coverage gain of a plurality of candidate next positions around the current position; select the candidate position with the maximum coverage gain as the next target to be moved, and drive the agent to the new target position; judge whether the termination condition is met, if not, start a new round of loop.

[0022] The technical scheme provided by the embodiments of the present disclosure can include the following beneficial effects: In the embodiments of the present disclosure, the three-dimensional structure information of the online construction environment is constructed by the above method without relying on the prior known point cloud or structure model. The method can be combined with the method based on an RGB camera and the method based on a depth sensor to improve the effect of coverage planning. The required three-dimensional model and initial observation path points can also be provided for the pre-set class method to further search for the shortest coverage path by using a TSP solver. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure. It is to be clearly understood that the drawings are merely representative of some embodiments of the present disclosure and that additional embodiments can be derived from the drawings without departing from the scope of the present disclosure.

[0024] Figure 1 A step diagram of an agent coverage planning method based on event stream depth prediction in an exemplary embodiment of the present disclosure is shown; Figure 2 A flowchart of an agent coverage planning method based on event stream depth prediction in an exemplary embodiment of the present disclosure is shown; Figure 3 A coverage path point result diagram of a coverage planning network model planning in an exemplary embodiment of the present disclosure is shown; Figure 4 A bridge photo taken in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0025] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations.

[0026] In addition, the accompanying drawings are only schematic and are non-limiting precise representations of the embodiments of the present disclosure. Identical components have been given the same reference numbers in the various figures, and so repeated description of these components has been omitted for clarity. Some of the blocks in the drawings are functionally divided into functional entities, which can not necessarily correspond to physically or logically separate entities.

[0027] An agent coverage planning method based on event stream depth prediction is provided in the present example implementation. Reference is made to Figure 1The agent coverage planning method based on event stream depth prediction can include the following steps, as shown in the following figure: Step S101: Construct a three-dimensional scene model in a virtual simulation environment, randomly sample a plurality of camera observation points, generate RGB images, true value depth information and camera observation poses, and take the true value depth information as a standard supervision signal; Step S102: According to each two adjacent RGB images, generate event stream data between the two shooting points; Step S103: Construct a coverage planning network model, wherein the coverage planning network model includes an event stream depth prediction network, a voxel occupancy prediction network and a coverage benefit prediction network; Step S104: Process the event stream data using the event stream depth prediction network to output a predicted depth map, take the depth map at the latest sampling time as a first supervision signal, and generate a point cloud set according to the depth map; Step S105: Process the point cloud set and the query point using the voxel occupancy prediction network to output voxel occupancy information, and generate a second supervision signal according to the voxel occupancy information at the query point; Step S106: Process the voxel occupancy information, the current camera observation pose and the historical camera observation pose using the coverage benefit prediction network to output the coverage gain of the current camera observation point, and generate a third supervision signal; Step S107: Sample M new camera points to be moved near the current camera point, and select the point with the maximum coverage gain as the next observation point; Step S108: Determine whether the coverage planning network model converges according to the standard supervision signal, the first supervision signal, the second supervision signal and the third supervision signal, if it converges, a trained coverage planning network model is obtained; wherein the standard supervision signal is used for a supervised learning method to make the event stream depth prediction network converge; the first supervision signal is used for training the depth prediction neural network module; the second supervision signal is used for training the voxel occupancy prediction network; and the third supervision signal is used for training the coverage benefit prediction network; Step S109: Apply the trained coverage planning network model to the agent, process the captured event stream, and select the point with the maximum coverage gain as the next observation point until the scene coverage task is completed or the termination condition is reached.

[0028] Through the above agent coverage planning method based on event stream depth prediction, the three-dimensional structure information of the environment is constructed online, and it does not depend on the prior known point cloud or structure model. It can be combined with methods based on RGB cameras and depth sensors to improve the effect of coverage planning. It can also provide the required three-dimensional model and initial observation path point for the pre-set method to further search for the shortest coverage path using a TSP solver.

[0029] Below, reference will be made to Figures 1 to 4 The various steps of the above-described agent coverage planning method based on event stream depth prediction in the present example embodiment will be described in more detail.

[0030] In step S101, a three-dimensional scene model is constructed in a virtual simulation environment, a number of camera observation points are randomly sampled, RGB images, ground truth depth information and camera observation poses are generated, and the ground truth depth information is used as a standard supervision signal.

[0031] Specifically, Isaac Sim virtual simulation software is used as a tool for generating a training data set. In this simulation environment, the three-dimensional model of the scene is mainly composed of a surface mesh and an appearance material. The appearance material is attached to the surface mesh and is used to generate the RGB image data of the camera observation bridge at a specific viewing angle. The mesh, after projection transformation, can be used as the benchmark ground truth of the three-dimensional structure information of the scene, and is used as the supervision signal (i.e., the standard supervision signal) of the event stream depth prediction network. Sampling points are randomly generated near the three-dimensional structure surface, and the RGB image information and ground truth depth information are captured using the functions provided by the virtual simulation software. The RGB image, depth image (i.e., ground truth depth information), and observation pose (i.e., camera observation pose) are recorded.

[0032] In step S102, event stream data between two adjacent RGB images is generated.

[0033] Specifically, for each two adjacent RGB images, a linear motion trajectory is constructed according to the shooting points, assuming that the agent motion speed amplitude at all adjacent points is the same. The agent is made to execute this trajectory, and the video sequence along the way is continuously collected. The video sequence is imported into Spikingsim to generate event stream observations between the two shooting points. And it is added to the data set generated in step S101.

[0034] In steps S103 to S107, a coverage planning network model is constructed, which includes an event flow depth prediction network, a voxel occupancy prediction network, and a coverage gain prediction network. The event flow depth prediction network processes the event flow data to output a predicted depth map, with the depth map at the latest sampling time serving as the first supervision signal, and a point cloud set is generated based on the depth map. The voxel occupancy prediction network processes the point cloud set and the query point to output voxel occupancy information, and a second supervision signal is generated based on the voxel occupancy information at the query point. The coverage gain prediction network processes the voxel occupancy information, the current camera observation pose, and the historical camera observation pose to output the coverage gain of the current camera observation point and generate a third supervision signal. M new camera points to be moved are sampled near the current camera point, and the point with the largest coverage gain is selected as the next observation point. The convergence of the coverage planning network model is determined based on the standard supervision signal, the first supervision signal, the second supervision signal, and the third supervision signal. If convergence is achieved, the trained coverage planning network model is obtained.

[0035] Specifically, a Transformer-based event stream deep prediction network model is established, with the model framework as follows: Figure 2 As shown, the model input is an event stream, and the output is a depth map. Here, we assume that the depth map of the event stream is derived from the depth map of the more recent sampling point in two predefined images, and use it as the supervision signal for the event stream depth prediction network. Assume that the existing point cloud information is... The point cloud is transformed from the depth map predicted by the event flow depth prediction network. The point cloud set formed by combining the point cloud information from two locations is Then there is.

[0036] (1) in, An event frame is a collection of event streams.

[0037] Construct a voxel occupancy prediction network. This module can implicitly represent whether a point in the scene is occupied or not. The values ​​are represented within a range, where 1 indicates that the location is occupied and 0 indicates that it is not occupied. The module will build a collection of location clouds. Camera pose Query location As input to the model, it then predicts a scalar value in the range [0,1], representing the current camera position. Below, the queried 3D location It is either occupied or not occupied. This application will include point cloud information. Four downsampling operations were performed, and the result of each downsampling operation was output to a self-attention module. In, and using multilayer perceptron Perform encoding mapping to obtain the output result. The self-attention network collects the K nearest neighbor point cloud of the current point p in each downsampled sample. As input, the network can encode voxel occupancy at different scales, thus supporting coverage planning for large-scale scenes. The supervision signal is the current camera position determined based on the ground-value point cloud information observable at the current camera position. Whether the signal is occupied.

[0038] (3) This step is mainly used to determine the currently known voxel occupancy information. and current camera observation point and historical camera observation points Information to predict the next given camera location The coverage gain module is used to improve the coverage of a specific scene. It first performs inverse sampling on the known voxel occupancy information to obtain a set of N discrete sampling points. Then, for each sampling point, its historical camera observation points are encoded into spherical harmonic information. The encoded information is then input into a Transformer framework to predict the coverage gain at each sampling point. Supervision information is still obtained from the ground-value 3D mesh. Finally, the coverage gains at all sampled points are aggregated into the overall coverage gain at the query camera point. After obtaining the gain prediction module, M new camera points to be moved are sampled near the current camera point each time. The point with the largest coverage gain is selected, as shown in Equation 4, and the camera is moved to that point. Extensive training is performed on both virtual environments and public datasets, with cross-validation conducted until the network module's performance meets the requirements.

[0039] (3) (4) In step S109, the trained coverage planning network model is applied to the agent, the captured event stream is processed, and the point with the largest coverage gain is selected as the next observation point until the scene coverage task is completed or the termination condition is met.

[0040] Specifically, the agent is equipped with a DVXplorer event camera. Since the event camera can only capture scene information when in motion, it is necessary to set the agent's movement speed and the proximity of each observation point. To save computation, the proximity is selected as the center points of the 6 faces and 8 vertices of a cube centered on the current position.

[0041] The trained event stream depth prediction network, including occupancy prediction and coverage gain prediction modules, predicts the next target camera position. During this process, it accumulates and updates local point cloud information, filters out depth predictions with large gradients, and uses smoothing filters to smooth the predicted depth information. It records the camera trajectory from the current camera position to the predicted camera position and integrates it into a historical camera position set.

[0042] During system initialization, a nearby location is randomly sampled from the current agent's location. In subsequent processes, the nearest location must be selected based on the coverage gain. After determining the target location, the agent is guided to that location, while simultaneously collecting and saving event stream information captured by the event camera.

[0043] Upon reaching the point, the agent stops and inputs the collected event stream data into the trained coverage planning network model, predicts local depth information and voxel occupancy in sequence, and then selects the point with the largest gain (as shown in formula (4)) as the point to move next based on the gain information of the nearby points predicted by the coverage gain module. Guide the agent to the obtained point, and perform the next round of sampling to obtain a nearby position, and continue until the coverage task is completed or the maximum time limit is reached.

[0044] In one specific embodiment, this embodiment evaluates the ShapeNet dataset, which contains 3D models of large-scale scenes. This study uses it to train the three prediction models mentioned above. Training was conducted on a desktop computer configured with an Intel Xeon Gold 6226R@3.90 GHz, Ubuntu 20.04, and ROS Noetic, with an Nvidia RTX A6000 graphics card and 48GB of video memory. A bridge scene was selected for coverage testing, and the results are as follows... Figure 3 and Figure 4 As shown. Among them, Figure 3 The result diagram of the coverage path points planned by the coverage planning network model; Figure 4These are photographs of the bridge taken during the process. A drone was set to move at a constant speed to generate an event stream. As can be seen from the image, the red coverage path points are distributed around the bridge. Depth maps are predicted from these points based on the event stream, and the next target point is planned using the occupancy prediction module and the coverage gain module. The 3D structure formed by stitching together the depth maps captured at each point basically completely reconstructs the 3D structure of the bridge, thus proving the effectiveness of this method.

[0045] Using the methods described above, this application proposes using an event camera, a novel sensor, for depth prediction to complete coverage planning. Because event cameras possess excellent characteristics such as high dynamic range, low power consumption, and immunity to motion blur, they can significantly overcome the shortcomings of existing methods. Compared to pre-defined coverage planning methods and methods that solve for coverage paths based on known point clouds, the method proposed in this application constructs the 3D structural information of the environment online, without relying on pre-known point clouds or structural models. This method can be combined with methods based on RGB cameras and depth sensors to improve the effectiveness of coverage planning. Furthermore, this application can provide the required 3D model and initial observation path points for pre-defined methods, allowing for further search of the shortest coverage path using the TSP solver.

[0046] It should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., in the above description indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this disclosure and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this disclosure.

[0047] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0048] In the embodiments of this disclosure, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0049] In embodiments of this disclosure, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0050] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0051] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. An agent coverage planning method based on event flow depth prediction, characterized in that, The method includes: A three-dimensional scene model is constructed in a virtual simulation environment. Several camera observation points are randomly sampled to generate RGB images, ground truth depth information and camera observation poses. The ground truth depth information is used as a standard supervision signal. Based on every two adjacent RGB images, generate event stream data between the two shooting locations; Construct a coverage planning network model, which includes an event flow depth prediction network, a voxel occupancy prediction network, and a coverage revenue prediction network. An event stream depth prediction network is used to process event stream data to output a predicted depth map. The depth map at the latest sampling time is used as the first supervision signal, and a point cloud set is generated based on the depth map. A voxel occupancy prediction network is used to process the point cloud set and the query point to output voxel occupancy information, and a second supervision signal is generated based on the voxel occupancy information at the query point. The voxel occupancy information, current camera observation pose, and historical camera observation pose are processed using a coverage gain prediction network to output the coverage gain of the current camera observation point and generate a third supervision signal. Sample M new camera locations to be moved near the current camera location, and select the location with the largest coverage gain as the next observation location; The standard supervision signal, the first supervision signal, the second supervision signal, and the third supervision signal are used to determine whether the coverage planning network model has converged. If it has converged, the trained coverage planning network model is obtained. The standard supervision signal is used to supervise the learning method so that the event flow deep prediction network converges. The first supervision signal is used to train the deep prediction neural network module. The second supervision signal is used to train the voxel occupancy prediction network. The third supervision signal is used to train the coverage gain prediction network. The trained coverage planning network model is applied to the agent, the captured event stream is processed, and the point with the largest coverage gain is selected as the next observation point until the scene coverage task is completed or the termination condition is met.

2. The agent coverage planning method based on event flow depth prediction according to claim 1, characterized in that, The steps of constructing a 3D scene model in a virtual simulation environment, randomly sampling several camera observation points, generating RGB images, ground truth depth information, and camera observation poses, and using the ground truth depth information as a standard supervision signal include: Construct a virtual simulation scene, import the 3D model of the virtual simulation scene, and set the camera parameters; Several camera observation points are randomly generated around the surface of the 3D model, and the camera observation poses are recorded. Capture the corresponding RGB image and ground-value depth information at each camera observation point; Use truth depth information as a standard monitoring signal.

3. The agent coverage planning method based on event flow depth prediction according to claim 2, characterized in that, The step of generating event stream data between two shooting points based on every two adjacent RGB images includes: For any two adjacent RGB images, the camera observation points are set to move from one point to another in a straight line at a constant speed. The agent in the simulation is controlled to execute the motion trajectory and continuously acquire video sequences; Import the video sequence into the event camera simulator to generate simulated event stream data between the two shooting locations.

4. The agent coverage planning method based on event flow depth prediction according to claim 3, characterized in that, The process of using an event stream depth prediction network to process event stream data and output a predicted depth map, with the depth map at the most recent sampling time serving as the first supervision signal, and generating a point cloud set based on the depth map, includes: The depth map of the event stream data is set as the depth map of the most recently sampled location in the two images, and it is used as the first supervision signal of the event stream depth prediction network; Set historical point cloud information as The current point cloud information is transformed from the depth map predicted by the event flow depth prediction network. ; Historical point cloud information and current point cloud information Point cloud set ; The point cloud set formed by combining the point cloud information from two locations is Then we have: in, An event frame is a collection of event stream data.

5. The agent coverage planning method based on event flow depth prediction according to claim 4, characterized in that, The steps of processing the point cloud set and query points using a voxel occupancy prediction network to output voxel occupancy information, and generating a second supervision signal based on the voxel occupancy information at the query point, include: Point cloud collection Four downsampling operations were performed to obtain point cloud sets with different levels of detail; Using self-attention modules Collect the K nearest neighbor point clouds of the current point location p in the point cloud set at each level of detail. Encoding nearest neighbor point cloud The features are used to encode and map voxel occupancy information using a multilayer perceptron. : in, Let p be the set of the K nearest point clouds. This aggregates the point cloud encoding results from four self-attention modules at four different sampling scales. This is the sparse global encoding for the entire current point cloud; Based on the ground truth point cloud information observed under the current camera pose, a second monitoring signal is generated to determine whether the current point is occupied.

6. The agent coverage planning method based on event flow depth prediction according to claim 5, characterized in that, The step of using a coverage gain prediction network to process voxel occupancy information, current camera observation pose, and historical camera observation pose to output the coverage gain of the current camera observation point and generate a third supervision signal includes: The voxel occupancy information was inversely sampled to obtain a set of N discrete sampling points. ; The relationship between sampling points and historical camera observation poses is encoded as spherical harmonic information. ; Use Transformer to aggregate all spherical harmonic information To predict the coverage gain at the current camera observation point: in, To obtain K points from the current perspective by performing an inverse sampling operation using occupancy information, This is an inverse sampling operation. The next perspective for network prediction Down Point coverage gain, For the next camera observation pose, To aggregate all Coverage gain at this viewpoint after the point The sum of occupancy rates For point Inverse depth from the current perspective; Using the ground truth depth information of the simulation environment, the true coverage gain of each candidate point is calculated to generate a third supervision signal.

7. The agent coverage planning method based on event flow depth prediction according to claim 6, characterized in that, The step of sampling M new camera locations near the current camera location and selecting the location with the largest coverage gain as the next observation location includes: Sample M new camera locations to be moved near the current camera location, and then select the location with the highest coverage gain. Move the agent to that location.

8. The agent coverage planning method based on event flow depth prediction according to claim 7, characterized in that, The steps for determining whether the coverage planning network model has converged based on the standard supervision signal, the first supervision signal, the second supervision signal, and the third supervision signal, and if it has converged, to obtain the trained coverage planning network model, include: The predicted values ​​of the event flow depth prediction network, voxel occupancy prediction network, and coverage gain prediction network are calculated based on the standard supervision signal, the first supervision signal, the second supervision signal, and the third supervision signal. The convergence is determined by the L2 norm of the difference between the predicted and supervised values ​​of the event flow depth prediction network, voxel occupancy prediction network, and coverage gain prediction network, and the rate of change of the L2 norm. If the rate of change of the L2 norm is less than a first preset threshold, the network converges; or if the L2 norm itself is less than a second preset threshold, the network converges. The coverage planning network model converges when the event flow depth prediction network, voxel occupancy prediction network, and coverage revenue prediction network all converge.

9. The agent coverage planning method based on event flow depth prediction according to claim 8, characterized in that, The process of applying the trained coverage planning network model to the agent, processing the captured event stream, and selecting the point with the largest coverage gain as the next observation point until the scene coverage task is completed or the termination condition is met includes: The agent moves from its current position to the next planned target point at a fixed speed. During this process, the event camera continues to work, collecting the event stream throughout the entire motion process; The collected event stream is input into the trained coverage planning network model to obtain the local depth map of the current endpoint location, and then converted into a local point cloud. The local point cloud is fused with the historical point cloud, and the global implicit occupancy map is updated using a voxel occupancy prediction network. The coverage gain prediction network receives the currently occupied map and all historical poses, and calculates the coverage gain for several candidate next points around the current location. Select the candidate point with the largest coverage gain as the next target to move to, and drive the agent to the new target point; Determine if the termination condition is met. If not, start a new cycle.

Images