A multi-agent sparse three-dimensional sampling point learning method and system

Abstract

The application discloses a multi-agent sparse three-dimensional sampling point learning method and system, extracts different scale feature maps of an image based on a backbone neural network in a computer vision neural network; extracts sampling features of sparse sampling points in multi-scale feature maps in a reverse manner based on the obtained different scale feature maps of the image; and predicts the positions of up-sampling sparse sampling points based on the sampling features of the sparse sampling points in the multi-scale feature maps, and selects sparse points through sampling importance predicted by a model. The application samples key points of an environment intelligently, can significantly reduce the amount of perception data through sparse perception, while retaining key environmental information, provides more accurate input for downstream tasks, and optimizes the calculation efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional sampling point learning, and particularly relates to a multi-agent sparse three-dimensional sampling point learning method and system. BACKGROUND

[0002] Traditional environment modeling relies on identifying specific objects in the environment, for example, traditional SLAM (Simultaneous localization and mapping) technology uses lidar sensors to measure environmental data, constantly adds and corrects point clouds to obtain a high-precision point cloud map, and uses loop detection state estimation and optimization to correct point cloud positions. However, this traditional method faces limitations in precision, feature extraction and matching, computational complexity, irregular and uncertain data processing, and other problems. Traditional SLAM usually assumes that the environment is static, but in real-world applications, the environment is often dynamic, such as moving pedestrians or vehicles that can interfere with the performance of the SLAM algorithm, leading to unstable map construction. As the space explored by robots or self-driving cars becomes larger, the amount of map data and computation involved in SLAM also increases rapidly.

[0003] Accurate perception modeling of the environment is the basis for achieving autonomous decision-making by robots. Robots, such as drones and self-driving cars, generally perceive the environment by collecting RGB images and then building a three-dimensional model. This process usually relies on uniform sampling within the environmental space. While this method can cover the entire environmental space, it results in a large amount of redundant information, especially in areas that are less relevant to the robot's movement. Since many areas in a three-dimensional environment do not contain key feature information, uniform sampling methods waste a large amount of computational resources and storage space, increasing the complexity of information processing.

[0004] The development of artificial intelligence technology provides a new way for robot perception. Deep learning can train end-to-end neural network models through large-scale data, directly predicting positions and environmental features from sensor data (such as camera images or point clouds) without the need for manually designed feature extraction algorithms, simplifying the entire mapping process. Based on the spatial structure and occupancy information obtained through three-dimensional perception, robots can better understand the geometric features and dynamic changes of their environment, thereby helping them navigate in complex and uncertain scenarios. In addition, three-dimensional perception provides a basis for real-time perception and reaction during navigation, enabling robots to effectively avoid obstacles and adapt to environmental changes, improving the safety and reliability of overall task execution. Therefore, in-depth research and optimization of three-dimensional perception technology is of great significance for improving the performance of robots in autonomous navigation and other related tasks.

[0005] There are two common methods in existing three-dimensional modeling technology solutions:

[0006] (1) Forward method, using forward mapping three-dimensional map construction method, the main steps include: Lift (dimension lifting): converting two-dimensional images from multiple cameras into point clouds in three-dimensional space. Specifically, LSS (Lift Splat Shoot) "lifts" each plane of pixels to multiple three-dimensional candidate points at different depths of each light ray through the internal and external parameter information of the camera, forming a three-dimensional space point cloud in the shape of a frustum. Splat (projection): spatial projection is performed on the generated three-dimensional point cloud, and these point clouds are projected into a predefined bird's eye view plane or three-dimensional grid voxel. Shoot (generation): finally, the bird's eye view is processed through convolution operation to predict the position, category and other information of the target. The entire process combines depth estimation and spatial projection to form a mapping from image to three-dimensional space, thereby obtaining a three-dimensional representation of the environment.

[0007] Its disadvantage is that if it does not rely on effective depth mapping, it is difficult to effectively estimate the thickness of the object in the invisible part. As shown in Figure 1 , the object thickness of the invisible part of each object relative to the observation position (center) is obviously distorted along the visible light direction.

[0008] (2) Backward method, three-dimensional perception method based on Transformer (transformer structure neural network) framework, which generates three-dimensional map through multi-view information, the core idea is to generate BEV (BirdEye's View, BEV) sampling points and corresponding Query (query vector) in space, and learn the weighted sampling mode of all positions in the current space through camera parameter mapping multi-view feature map to form a stable bird's eye view representation. Multi-view fusion: BEVFormer (BEV view based transformer neural network modeling method) first obtains multi-view two-dimensional images from different cameras, and uses neural network to extract features of each image. Bird's eye view query: through the query mechanism, BEVFormer can extract bird's eye view related features from image features to construct three-dimensional map representation under bird's eye view. Compared with the forward method, the advantage of this method is its stronger feature fusion capability, especially in irregular scenes or complex dynamic environments.

[0009] Its disadvantage is that obtaining the features of each BEV grid needs to be uniformly sampled from multi-view images, without distinguishing between effective and ineffective spaces, which reduces the sampling efficiency of the model, and ineffective sampling points may also increase the noise of the model.

[0010] (3) Sparse sampling modeling method is a sparse improvement of the backward method. The main idea of this method is to set the maximum sampling number limit in the fixed space to limit the model size, and to use the coarse-to-fine method to continuously refine the perception output accuracy. The design steps are: initially multiple large-scale perception areas in the space; sparse sampling prediction step by step to refine the space occupation situation and discard the idle area; the process of continuous refinement obtains a sparse occupied space representation.

[0011] The disadvantage is that in the uniform sampling method of continuously refining the occupied space, the specific up-sampling mode still needs to be based on the hand-designed sampling method in the sparse space, and the designed sparse process cannot recover the accumulated error.

[0012] The two basic methods introduced above can establish a general perception map: bird's eye view or occupancy grid map. However, in three-dimensional environment modeling, most of the space does not contain valuable information, and only a small part of the area needs to be accurately modeled. Both methods use the same calculation method for empty areas and occupied areas, resulting in meaningless calculations in the model. SUMMARY

[0013] The technical problem to be solved by the present application is to provide a multi-agent sparse three-dimensional sampling point learning method and system to solve the technical problem of wasting a large amount of computing resources and storage space and increasing the complexity of information processing in the prior art.

[0014] The technical scheme adopted by the present application is as follows:

[0015] A multi-agent sparse three-dimensional sampling point learning method, comprising the following steps:

[0016] Extracting different scale feature maps of images based on the backbone neural network in the computer vision neural network;

[0017] Based on the obtained different scale feature maps of images, extracting sampling features of sparse sampling points in multi-scale feature maps by a backward method;

[0018] Predicting the positions of up-sampling sparse sampling points based on the sampling features of sparse sampling points in multi-scale feature maps, and selecting sparse points by the sampling importance predicted by the model.

[0019] Preferably, the extraction of different scale feature maps of images is specifically:

[0020] The multi-view RGB image is taken as input, and the feature pyramid network is used to uniformly output different scale feature maps with a specific number of channels.

[0021] Preferably, the sparse sampling points are extracted in the reverse manner in the sampling features of the multi-scale feature map, and the sampling features of the multi-scale feature map are specifically as follows:

[0022] The sparse sampling points are defined in the map coordinates in the real world coordinate system.

[0023] The sparse sampling points are calculated by obtaining the camera intrinsic parameters of each image and the transfer matrix of the camera relative to the map coordinates, and the pixel coordinates of the sampling points on the RGB image are calculated.

[0024] The pixel coordinates of the sampling points on the RGB image are scaled to the feature image pixel coordinates according to the size of the original image relative to the multi-scale feature map, and the sampling point coordinates are obtained by sampling on the multi-scale feature map.

[0025] Preferably, the sparse points are selected by the sampling importance predicted by the model, and the sampling importance predicted by the model is specifically as follows:

[0026] The original sampling point features and the up-sampling point features are interacted through a neural network to obtain the up-sampling point weight; the rasterization of the up-sampling point with the weight is used to obtain the raster map, and the weight prediction network and the overall network are updated by using the supervised raster map output and the back propagation.

[0027] Preferably, the weight prediction network and the overall network are updated by using the back propagation, and the weight prediction network and the overall network are updated by using the back propagation.

[0028] In the target perception space, the sampling points are initialized in a uniform distribution.

[0029] The positions of the map coordinates of the sparse sampling points in the image pixel coordinates are calculated by the camera intrinsic parameters and the transfer matrix of the map to the camera sensor, and the local features of the positions are fused as the sampling point features.

[0030] A plurality of offsets are predicted for each coefficient sampling point by using a neural network to form up-sampling points.

[0031] A three-dimensional grid network is used to assign each point cloud to the nearest grid voxel.

[0032] The grid features are decoded into a three-dimensional grid map by using a neural network, and the map loss is supervised by using a cross-entropy to update the network parameters by back propagation.

[0033] According to the artificial set hyperparameters, it is judged whether the offset learning and up-sampling of the sampling points need to be continued to realize the learning of the effective area dense but overall sparse sampling mode.

[0034] Preferably, the up-sampling points are specifically as follows:

[0035] The position features of the original sampling points and the up-sampling points are interacted by using a neural network to obtain the up-sampling point features, and the sampling weights of the up-sampling points are predicted by using the point features aggregated with the global features ; in the iteration process, the sampling data tends to be multiple of the probability weight.

[0036] Preferably, the feature of each grid voxel is :

[0037]

[0038] wherein, is the sampling feature of the sampling point mapped to the image space, is the number of sampling points falling within the current voxel space, is the importance value of the sampling point predicted by the neural network.

[0039] Preferably, the cross-entropy is calculated as follows:

[0040]

[0041] wherein, is the number of categories, is the th real label, is the th model prediction output.

[0042] Preferably, if further upsampling is needed, the weight of the sparse sampling point will be converted into a relative sampling probability value by an activation function, and the sampling point with a larger weight will be selected according to the probability for further upsampling processing.

[0043] In a second aspect, an embodiment of the present application provides a multi-agent sparse three-dimensional sampling point learning method system, comprising:

[0044] An extraction module extracts different scale feature maps of an image based on a backbone neural network in a computer vision neural network;

[0045] A reverse module extracts sampling features of sparse sampling points in multi-scale feature maps in a reverse manner based on the obtained different scale feature maps of the image;

[0046] An output module predicts the positions of the up-sampled sparse sampling points based on the sampling features of the sparse sampling points in the multi-scale feature maps, and selects the sparse points according to the sampling importance predicted by the model.

[0047] In a third aspect, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above multi-agent sparse three-dimensional sampling point learning method when executing the computer program.

[0048] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium comprising a computer program, which, when executed by a processor, implements the steps of the multi-agent sparse 3D sampling point learning method described above.

[0049] In a fifth aspect, a chip comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-agent sparse 3D sampling point learning method described above when executing the computer program.

[0050] In a sixth aspect, an embodiment of the present application provides an electronic device comprising a computer program, which, when executed by the electronic device, implements the steps of the multi-agent sparse 3D sampling point learning method described above.

[0051] Compared with the prior art, the present application has at least the following beneficial effects:

[0052] A multi-agent sparse 3D sampling point learning method, by learning the importance of sampling, realizes dense sampling in important areas, and sets the number of sparse sampling points to determine the computational complexity of the model; greatly reduces the number of data points that need to be processed, reduces the computational complexity and operation time, and improves the efficiency of processing large-scale 3D data sets; focusing on dense sampling in important areas can more accurately capture and understand the key information of the scene, thereby improving the accuracy of computer vision tasks such as 3D reconstruction, object recognition, and scene understanding; through sparse sampling, the required storage space can be reduced, and through learning sampling, the model can better identify which features are important for completing a specific task, and can improve the model's generalization ability to new scenes or data; in a multi-agent system, each agent is responsible for sampling different parts or aspects of the data, and then shares the learned sampling importance information to improve the performance and collaboration efficiency of the overall system; the number of sparse sampling points can be adjusted according to different application scenarios and requirements, so that the model can maintain optimal performance under different conditions; in the fields of virtual reality, autonomous vehicle perception, and environmental monitoring, sparse sampling technology can be used to achieve more efficient and accurate application solutions.

[0053] Further, the design structure of the pyramid is used to unify the multi-scale feature map channels, and the sampling points can be mapped to the same dimension features in the multi-scale feature space to obtain the aggregated sampling point features at different scales.

[0054] Further, the spatial sampling points are mapped to the pixel space points using camera parameters. By sampling the pixels around the sampling points on the multi-scale feature maps with the same channel, the sampling point features are obtained. The process of obtaining features in this way is the same matrix operation method, which is conducive to the parallel ability of the device.

[0055] Further, the obtained sampling points can obtain global sampling information through the interaction of the neural network. The neural network uses the global sampling information and the sampling point features as inputs to predict the importance of the sampling points.

[0056] Further, the importance of the sampling points is mapped to a sampling probability value through an activation function, and the sampling point features in each voxel are weighted and averaged to obtain the voxel features through the sampling probability. According to claim 8, the voxel feature learning outputs a semantic voxel map, and the loss is calculated by using a loss function and the sampling probability is updated by using a gradient descent method.

[0057] Further, if further upsampling is needed, the weight of the sparse sampling points is converted into a relative sampling probability value through an activation function, and the sampling points with larger weights are selected according to the probability for further upsampling. The adaptive sampling density can be obtained through the probability sampling learned by the network. Compared with the fixed large sampling probability in the set important area, the adaptive probability can avoid the deterioration of the ratio of the number of sampling points in the important sampling area to the number of sampling points in the unimportant sampling area in the multiple upsampling process. For example, if the sampling probability ratio of the important area to the unimportant area is 2:1, then after the first upsampling, the ratio of the number of important points to the number of unimportant points is 2:1, and after the second upsampling, the ratio is 4:1. In this way, it will be difficult to effectively perceive and model the unimportant area in the deep neural network for modeling a large range of environment. The self-learning probability can consider the current sampling density and adaptively predict the sampling probability in the important area through the interaction of the neural network, which is more suitable for the universal three-dimensional scene perception modeling.

[0058] It can be understood that the beneficial effects of the above-mentioned second aspect to the sixth aspect can be referred to the related description in the first aspect, which will not be repeated here.

[0059] In summary, the present application can significantly reduce the amount of perception data while retaining key environmental information by intelligently selecting key points in the environment for sampling, and can provide more accurate input for downstream tasks and optimize computing efficiency.

[0060] The technical solutions of the present application will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The object thickness prediction is severely distorted;

[0062] Figure 2 The present application is a flowchart;

[0063] Figure 3 The present application is a system block diagram;

[0064] Figure 4 The present application is a flowchart of the learnable probability sampling;

[0065] Figure 5 A schematic diagram of a computer device according to an embodiment of the present application is provided.

[0066] Figure 6 A block diagram of a chip according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0068] In the description of the present application, it should be understood that the terms "comprise" and "include" indicate the presence of described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0069] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, unless otherwise clearly indicated by the context, the singular forms "a", "an" and "the" are intended to include the plural forms as well.

[0070] It should be further understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations, for example, A and / or B can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " in the present application generally represents an "or" relationship between the front and rear associated objects.

[0071] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, the first preset range can also be referred to as the second preset range, and similarly, the second preset range can also be referred to as the first preset range without departing from the scope of the embodiments of the present application.

[0072] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "when it is determined" or "in response to determining" or "when [a stated condition or event] is detected" or "in response to detecting [a stated condition or event]."

[0073] Various structural diagrams according to the disclosed embodiments of the present application are shown in the accompanying drawings. These diagrams are not drawn to scale in which certain details are exaggerated for clarity of presentation and may omit certain details. The shapes and relative sizes and positions of the various regions, layers, and their relative positions shown in the drawings are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations, and regions / layers with different shapes, sizes, and relative positions can be additionally designed by those skilled in the art according to actual needs.

[0074] The present application provides a multi-agent sparse three-dimensional sampling point learning method, which restores a dense three-dimensional environment representation model from limited point clouds, reduces model resource occupation, improves the efficiency and generalization ability of the model by reducing redundant parameters and unnecessary calculations in the model, and simplifies the model structure by selectively retaining parameters or features essential to the task. Not only can the calculation complexity and memory occupation be reduced, but also the inference speed can be accelerated and resource consumption can be reduced.

[0075] Please refer to Figure 2 The multi-agent sparse three-dimensional sampling point learning method of the present application comprises the following steps:

[0076] S1, extracting different scale feature maps of an image based on a backbone neural network in a computer vision neural network;

[0077] The backbone neural network in the computer vision neural network is usually referred to as the main feature extractor part, and a multi-view RGB image is used as input and a feature image is used as output.

[0078] The input image is obtained through a convolutional neural network to obtain feature maps of different scales; the number of channels of these feature maps is much larger than the 3 channels (R, G, B) of the RGB image, and is used to represent the neural network abstract features at this position;

[0079] The different scale feature maps output by the backbone neural network usually have different numbers of channels. These multi-scale features together represent the feature extraction of the network, but different numbers of channels are not suitable for uniform matrix operations. Therefore, a feature pyramid network (Feature Pyramid Networks) is used to unify the multi-scale features to the same number of channels for sampling points to sample different scale features.

[0080] S2, based on the obtained image different scale feature map, the sparse sampling points are extracted in the sampling feature of the multi-scale feature map in a reverse manner;

[0081] The sparse sampling points are defined in a certain coordinate system in the real world, which is temporarily referred to as a map coordinate;

[0082] The sparse sampling points are calculated by obtaining the camera intrinsic parameter of each image and the transfer matrix of the camera relative to the map coordinate, and the pixel coordinates of the sampling points on the RGB image are calculated.

[0083] The pixel coordinates are scaled to the feature image pixel coordinates according to the original image relative to the feature map size, and the sampling point coordinates are obtained by sampling on the multi-scale feature map.

[0084] S3, the positions of the up-sampled sparse sampling points are predicted based on the sampling features of the sparse sampling points in the multi-scale feature map, and the sparse points are selected through the sampling importance predicted by the model.

[0085] The current sparse sampling points are output by a neural network, the positions of the up-sampled sparse sampling points are predicted, and the output modeling accuracy is increased; the original sampling point features and the up-sampled point features are interacted through a neural network to obtain the up-sampled point weight; the grid map is obtained by using the gridization of these point clouds with weights, the grid map output is supervised, and the weight prediction network and the overall network are updated through back propagation.

[0086] Referring to Figure 4 , the application adopts a learnable probability sampling mode, which involves sampling point weight learning and a sampling method based on weight probability, and the specific steps are as follows:

[0087] S301, initializing uniformly distributed sampling points in a target perception space;

[0088] Although the initial sampling points are uniformly distributed, the sampling density is significantly lower than the density required by the final output.

[0089] S302, the map coordinates of the sparse sampling points are calculated in the image pixel coordinates through the camera intrinsic parameter and the transfer matrix of the map to the camera sensor, and the local features of the position are fused as the sampling point features;

[0090] S303, a plurality of offsets are predicted for each coefficient sampling point by a neural network to form up-sampled points;

[0091] The position features of the original sampling points and the up-sampled points are interacted through a neural network to obtain the up-sampled point features, and the sampling weights of the up-sampled points are predicted through a linear layer ; In the iteration process, the sampling times of the model are very large, so the sampling data of this layer in the iteration process will tend to be multiple of the probability weight.

[0092] S304, each point cloud is assigned to the nearest grid voxel using a three-dimensional grid network;

[0093] Feature gridding is performed, and the features of all points inside are summed by weighting to obtain the feature of each grid voxel For:

[0094]

[0095] wherein, is the sampling feature of the sampling point mapped to the image space, is the number of sampling points falling within the current voxel space, is the importance value of the sampling point predicted by the neural network.

[0096] S305, the grid features are decoded into a three-dimensional grid map by a neural network, and a map loss is output by cross-entropy supervision, and the network parameters are updated by back propagation;

[0097] There are c categories, is the true label (a vector of c dimensions in one-hot encoding form), if the sample belongs to the k (k≤c) class, the k-1th position of the vector is 1 and the rest is 0. represents the model prediction output, the dimension and the vector are the same, and each position represents the probability prediction of the corresponding category.

[0098] Cross-entropy is calculated as follows:

[0099]

[0100] wherein, is the number of categories, is the kth real label, is the kth model prediction output. S306, according to the artificially set hyperparameter (neural network depth), it is judged whether the offset learning and upsampling of the sampling points need to be continued to realize the learning of the effective area dense but overall sparse sampling mode;

[0101] S307, if the upsampling needs to be continued, the weight of the sparse sampling point will be converted into a relative sampling probability value by an activation function, and the sampling point with larger weight is selected according to the probability for further upsampling processing.

[0102] S307, if the upsampling needs to be continued, the weight of the sparse sampling point will be converted into a relative sampling probability value by an activation function, and the sampling point with larger weight is selected according to the probability for further upsampling processing. ​

[0103] In another embodiment of the present application, a multi-agent sparse three-dimensional sampling point learning method system is provided, which can be used to implement the multi-agent sparse three-dimensional sampling point learning method described above. Specifically, the multi-agent sparse three-dimensional sampling point learning method system includes an extraction module, a reverse module, and an output module.

[0104] The extraction module extracts different scale feature maps of an image based on a backbone neural network in a computer vision neural network.

[0105] The reverse module extracts sampling features of sparse sampling points in multi-scale feature maps in a reverse manner based on the obtained different scale feature maps of the image.

[0106] The output module predicts the positions of up-sampled sparse sampling points based on the sampling features of the sparse sampling points in the multi-scale feature maps, and selects sparse points through the sampling importance predicted by the model.

[0107] In another embodiment of the present application, a terminal device is provided, which includes a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement a corresponding method flow or a corresponding function. The processor described in the embodiments of the present application can be used for the operation of the multi-agent sparse three-dimensional sampling point learning method, including:

[0108] extracting different scale feature maps of an image based on a backbone neural network in a computer vision neural network, extracting sampling features of sparse sampling points in multi-scale feature maps in a reverse manner based on the obtained different scale feature maps of the image, and predicting the positions of up-sampled sparse sampling points based on the sampling features of the sparse sampling points in the multi-scale feature maps, and selecting sparse points through the sampling importance predicted by the model.

[0109] Please refer to Figure 5 , the terminal device is a computer device, the computer device 60 of the embodiment includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61, and the computer program 63 implements the multi-agent sparse three-dimensional sampling point learning method in the embodiment when executed by the processor 61. To avoid repetition, this will not be repeated here. Alternatively, the computer program 63 implements the functions of each model / unit in the multi-agent sparse three-dimensional sampling point learning method system when executed by the processor 61. To avoid repetition, this will not be repeated here.

[0110] The computer device 60 can be a desktop computer, a notebook, a palm computer, and a cloud server, etc. The computer device 60 can include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art can understand that Figure 5 is only an example of the computer device 60 and does not constitute a limitation on the computer device 60, and can include more or fewer components than shown, or combine certain components, or different components, for example, the computer device can also include an input / output device, a network access device, a bus, etc.

[0111] The processor 61 can be a central processing unit (CPU), and can also be other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0112] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or a memory of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0113] Further, the memory 62 can include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0114] Referring to Figure 6 The terminal device is a chip, and the chip 600 of the embodiment includes one or more processors 622 and a memory 632 for storing computer programs executable by the processor 622. The computer programs stored in the memory 632 can include one or more than one module each corresponding to a set of instructions. In addition, the processor 622 can be configured to execute the computer programs to perform the multi-agent sparse three-dimensional sampling point learning method described above.

[0115] In addition, the chip 600 can further include a power supply component 626 configured to perform power management of the chip 600 and a communication component 650 configured to implement communication of the chip 600, such as wired or wireless communication. In addition, the chip 600 can further include an input / output interface 658. The chip 600 can operate based on an operating system stored in the memory 632.

[0116] In another embodiment of the present application, the present application also provides a storage medium, specifically a computer readable storage medium, which is a memory device in a terminal device and is used to store programs and data. It can be understood that the computer readable storage medium herein can include an internal storage medium in the terminal device, and of course can also include an expansion storage medium supported by the terminal device. The computer readable storage medium provides a storage space, and the storage space stores an operating system of the terminal. In addition, one or more than one instruction suitable for being loaded and executed by a processor is also stored in the storage space, and the instruction can be one or more than one computer program. It should be noted that the computer readable storage medium herein can be a high-speed RAM memory or a non-volatile memory such as at least one disk memory.

[0117] The one or more than one instruction stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the multi-agent sparse three-dimensional sampling point learning method in the above embodiments; the one or more than one instruction in the computer readable storage medium is loaded and executed by the processor as follows:

[0118] Different scale feature maps of an image are extracted based on a backbone neural network in a computer vision neural network; sampling features of sparse sampling points in multi-scale feature maps are extracted in a reverse manner based on the obtained different scale feature maps of the image; positions of the up-sampled sparse sampling points are predicted based on the sampling features of the sparse sampling points in the multi-scale feature maps, and the sparse points are selected through sampling importance predicted by the model.

[0119] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art without creative labor on the basis of the embodiments in the present application belong to the scope of protection of the present application.

[0120] Please refer to Figure 3 The present application includes two parts of 2D RGB image processing and 3D perception modeling based on point cloud recovery grid, specifically as follows:

[0121] 2D image processing is a common neural network processing method for computer vision: the image is processed through CNN / Transformer (convolutional / transformer) vision network to obtain different scale features at different depths.

[0122] 3D perception modeling (innovative part): sparse sampling points obtain their own sampling point feature data by acquiring multi-scale feature map data;

[0123] The point-based neural network model is used to predict the importance of the point cloud, and the grid voxel features are obtained by weighted summation using the importance values after the point cloud is rasterized;

[0124] The grid map is used as supervision to update the weight network; the model point cloud is increased and the perception output is refined according to the weight up-sampling.

[0125] The present application has improved the existing backward model through sparse sampling to realize a simplest prototype, and under the condition that the model memory is reduced by nearly 40%, the index still reaches 87% of the original model.

[0126]

[0127] In summary, the multi-agent sparse three-dimensional sampling point learning method and system provided by the application has a learnable sparse sampling mode based on probability, and compared with a fixed sampling model, higher sampling flexibility is achieved by using fewer sampling points.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for easy distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the system can be referred to the corresponding process in the foregoing method embodiments, which will not be described here.

[0129] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0130] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0131] In the embodiments provided by the application, it should be understood that the disclosed devices / terminals and methods can be implemented by other ways. For example, the device / terminal embodiments described above are only schematic, and for example, the division of the modules or units is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0133] In addition, each functional unit in various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0134] The integrated module / unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0135] The present application is described with reference to flowcharts and / or block diagrams according to the method, equipment and computer program product of the embodiment. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing equipment to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment produce a machine that implements the flowchart and / or block diagram. Figure 1 The flowchart and / or block diagram Figure 1means for performing the function specified in the block or blocks.

[0136] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 one or more flowcharts and / or blocks Figure 1 means for performing the function specified in the block or blocks.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 one or more flowcharts and / or blocks Figure 1 means for performing the function specified in the block or blocks.

[0138] The above merely illustrates the technical idea of the present application, and cannot be used to limit the protection scope of the present application. Any modification made according to the technical idea of the present application, on the basis of the technical scheme, falls within the protection scope of the claims of the present application.

Claims

1. A multi-agent sparse three-dimensional sampling point learning method, characterized in that: The following steps are involved: Extract feature maps of different scales of images based on the backbone neural network in computer vision neural network; Based on the obtained feature maps of different scales of the image, the sampling features of the sparse sampling points in the multi-scale feature map are extracted in a reverse manner; The positions of sparse sampling points are sampled based on the sampling feature prediction of the multi-scale feature map, and the sparse points are selected based on the sampling importance predicted by the model, specifically: The original sampling point features and the upsampled point features are interacted through a neural network to obtain the upsampled point weights; the grid map is obtained by rasterizing the weighted upsampled points, and the weight prediction network and the overall network are updated by supervising the raster map output and back propagation.

2. The multi-agent sparse three-dimensional sampling point learning method according to claim 1, characterized in that: Extracting feature maps of different scales of images is as follows: Taking multi-view RGB images as input, the feature pyramid network uniformly outputs feature maps of different scales with a specific number of channels.

3. The multi-agent sparse three-dimensional sampling point learning method according to claim 1, characterized in that: The sampling features of sparse sampling points in the multi-scale feature map are extracted in a reverse manner as follows: Define sparse sampling points as map coordinates in the real world coordinate system; The sparse sampling points are calculated by obtaining the camera intrinsic parameters of each image and the transfer matrix of the camera relative to the map coordinates to calculate the pixel coordinates of the sampling points on the RGB image; The sampling point scales the pixel coordinates of the sampling point on the RGB image to the pixel coordinates of the feature map according to the size of the original image relative to the multi-scale feature map, and samples the sampling point on the multi-scale feature map to obtain the coordinates of the sampling point.

4. The multi-agent sparse three-dimensional sampling point learning method according to claim 1, characterized in that: Using back propagation to update the weight prediction network and the overall network is as follows: Initialize uniformly distributed sampling points in the target perception space; The position of the map coordinates of the sparse sampling points in the image pixel coordinates is calculated through the camera intrinsic parameters and the transfer matrix from the map to the camera sensor, and the local features of the position are fused into the sampling point features; Use neural networks to predict multiple offsets for each current coefficient sampling point to form upsampling points; Using a three-dimensional grid network, each point cloud is assigned to the nearest grid voxel; The grid features are decoded into a three-dimensional grid map through a neural network, and the output map loss is supervised by cross entropy, and back propagation is performed to update the network parameters; Based on the manually set hyperparameters, it is determined whether it is necessary to continue the offset learning and upsampling of the sampling points to achieve the learning of a sampling pattern with dense effective areas but sparse overall.

5. The multi-agent sparse three-dimensional sampling point learning method according to claim 4, characterized in that: The specific upsampling points are: The neural network is used to interact the position features of the original sampling points and the up-sampling points to obtain the up-sampling point features, and the sampling weights of the up-sampling points are predicted by aggregating the point features of the global features. ; During the iteration process, the sampled data tends to be a multiple of the probability weight.

6. The multi-agent sparse three-dimensional sampling point learning method according to claim 4, characterized in that: Characteristics of each raster voxel for: in, is the sampling feature that maps the sampling points to the image space, is the number of sampling points falling within the current voxel space, It is the importance value of the sampling point predicted by the neural network.

7. The multi-agent sparse three-dimensional sampling point learning method according to claim 4, characterized in that: Cross Entropy The calculation is as follows: in, is the number of categories, For the True labels, For the The model predicts the output.

8. The multi-agent sparse three-dimensional sampling point learning method according to claim 4, characterized in that: If further upsampling is required, the weights of the sparse sampling points will be converted into relative sampling probability values ​​through the activation function, and the sampling points with larger weights will be selected for further upsampling based on the probability.

9. A multi-agent sparse three-dimensional sampling point learning method system, characterized in that: include: The extraction module extracts feature maps of different scales of the image based on the backbone neural network in the computer vision neural network; The reverse module extracts the sampling features of sparse sampling points in the multi-scale feature map in a reverse manner based on the obtained feature maps of different scales of the image; The output module samples the locations of sparse sampling points based on the sampling feature prediction of the multi-scale feature map and selects sparse points based on the sampling importance predicted by the model. Specifically: The original sampling point features and the upsampled point features are interacted through a neural network to obtain the upsampled point weights; the grid map is obtained by rasterizing the weighted upsampled points, and the weight prediction network and the overall network are updated by supervising the raster map output and back propagation.

Images