Communication and positioning integration method for intelligent machine group

Abstract

The invention discloses a communication and positioning integration method for an intelligent machine group. The method is used for cooperation of multiple intelligent robots. The method comprises the steps that an intelligent robot constructs a query vector from radar point cloud data sensed currently through medium-term feature extraction and interaction module points, space importance information is calculated, and medium-term features are extracted for communication interaction; the base station clusters the intelligent robots through an optimal communication link construction module, allocates optimal communication links among the robots in clusters, and complements communication links of boundaries among the clusters; and the intelligent robot processes a middle-term feature time sequence sent by the cooperative positioning object and a feature map sequence collected by the intelligent robot through a time sequence relative positioning module, and the relative position relation between the robots is obtained. According to the method, the utilization of time sequence information is considered, the real-time operation under the limitation of objective communication conditions is realized, the cross-device features are fused in the space-time dimension to realize efficient and high-precision continuous cooperative positioning, and the registration precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-intelligent robot collaboration, and specifically relates to a method for integrating communication and positioning of a group of intelligent machines. Background Art

[0002] In the field of intelligent robots, in order for robots to perform tasks safely, reliably and with high accuracy, they are often required to have high positioning accuracy in the corresponding environment. The current robot positioning method is mainly based on the SLAM (Simultaneous Localization and Mapping) algorithm based on high-precision maps. It pre-builds a high-precision map of the scene and performs map-level calculations and re-positioning based on real-time perception data during operation.

[0003] In order to achieve the above goals, due to the unavoidable problems at the device level such as the difficulty in meeting the accuracy requirements or the accumulated errors of commonly used GPS and inertial measurement units (IMUs), the current common practice is usually based on high-precision maps, that is, by matching sensor data with pre-built high-precision maps to obtain absolute coordinates in the map coordinate system, such as longitude and latitude. However, although the positioning solution of pre-built maps can help improve the speed and accuracy of positioning, this solution that relies entirely on maps means that the robot's collaborative algorithm cannot run directly without maps. When the deviation between the pre-built map and the distribution of perception data is in high-interference scenarios such as underground, mines, and mountainous areas, it is difficult for intelligent robots to carry out high-reliability operations under such conditions due to the limitations of lighting, rapid environmental changes, and insufficient positioning device accuracy, which limits their application scenarios.

[0004] In the scenario of multi-robot group interconnection and collaboration, the multi-robot system operation algorithm usually pays more attention to the relative position relationship between robots rather than their absolute position in the global coordinate system. Only the relative position relationship between the neighbor robot and the main robot is needed to complete the conversion and alignment of the coordinate system, and then complete the intelligent decision-making process from the perspective of the main robot. Therefore, the collaborative localization algorithm that is independent of high-precision maps has received extensive attention and research in the field of autonomous driving. Current related work focuses on registering the point clouds of both parties, and deriving the rotation matrix between the coordinate systems of the two point clouds to achieve relative positioning without maps. However, this type of work lacks consideration of dynamic operating conditions, that is, in the continuous positioning scenario targeted by the present invention, the point cloud data shared in the collaborative perception process between robots has temporal correlation, and the corresponding relative position relationship also has this relationship, but the existing work lacks consideration of temporality, and it mainly completes the registration of multi-angle point clouds of a single frame. In the process of continuous relative positioning, due to the strong temporal correlation between the previous and next point cloud frames and the relative positions, the continuous temporal information that is strongly correlated with the relative position is usually ignored. On this basis, in order to ensure the reliability of continuous positioning, a large amount of computing and communication resources are often wasted. In addition, the current point cloud registration algorithm is usually implemented based on the original point cloud data of both parties. Although this information contains the largest amount of information, due to the excessively large amount of point cloud data, it will be difficult to avoid the problem of causing a large amount of communication resources in actual scenarios, and it is difficult to ensure the real-time nature of collaborative positioning, which limits its availability in real scenarios.

[0005] On April 12, 2024, the Chinese invention patent with publication number CN117870652A disclosed a land-air online collaborative positioning method based on radar, inertial, and visual fusion. In the process of relative positioning, both the UAV and the unmanned vehicle realize their own positioning through the fusion of airborne radar and inertial units. At the same time, the relative position of the unmanned vehicle is observed based on the UAV vision, and then the relative pose estimation is realized by the fusion and registration of the UAV and the unmanned vehicle point cloud. However, this technical solution has too many requirements for the categories of perception data. In addition to radar point cloud data, it requires multiple types of perception data sources and land-air collaborative equipment assistance to complete highly stable relative positioning. Although sufficient and rich perception data can bring higher accuracy to collaborative positioning, it often needs to be weighed against the communication and computing load of the equipment and environment in real scenarios. With the increase in the number of devices in the system, the excessive perception data demand means an increase in system cost on the one hand, and a nonlinear increase in the peak communication and computing resource usage in the system on the other hand, which is often unacceptable in actual scenarios.

[0006] The Chinese invention patent application with publication number CN117156415A disclosed a communication resource management method for vehicle network perception collaboration on December 1, 2023. The vehicle speed is estimated by using the position obtained by wireless perception, and the vehicle position at the next moment is predicted based on the perception estimation model. The clustering method is used based on the prediction results to cluster the vehicles corresponding to the vehicle user terminals. The system throughput is calculated based on the results of clustering management and the achievable communication rate. The resource allocation optimization problem is constructed with maximizing the system throughput as the optimization goal. The management method of communication resources in this technical solution takes maximizing the system throughput as the optimization goal. The clustering method only depends on the location of the device. However, in scenarios where there may be a large number of invalid communications, although the optimization method of maximizing throughput can make full use of resources, it ignores the attention and management of invalid communications, which will make it difficult for indicators to directly reflect the problem of waste of communication resources.

[0007] The Chinese invention patent application with publication number CN117930219A disclosed a method for positioning swarm drones based on millimeter-wave radar on April 26, 2024. It is aimed at scenarios with navigation failure, and uses millimeter-wave radar technology to solve the problem of positioning swarm drones. It estimates the expected value of relative position based on the cluster state before failure, and then combines the radar and inertial measurement unit of a single device to complete relative positioning. However, this technical solution is based on the premise of the navigation failure scenario, and relies on the cluster position and motion relationship before the navigation failure for positioning. For scenarios without high-precision maps in the initial state or relative positioning starting from the initial state, since there is no channel to obtain the initial expected value, this technical solution is difficult to adapt to such scenarios.

[0008] On October 31, 2023, the Chinese invention patent application with publication number CN116977963A disclosed a multimodal collaborative perception method and system for autonomous driving based on the BEV perspective. In order to solve the problem of poor collaborative perception effect of multimodal data in the fusion process of autonomous driving, the technical solution extracts point cloud BEV features, image BEV features and depth BEV features from the acquired original point cloud data and multi-view camera image data, and then fuses each BEV feature to obtain multimodal BEV fusion features. The multimodal BEV fusion features are processed by a multi-task head to obtain three-dimensional target detection results and BEV perspective semantic segmentation results. The technical solution uses point cloud and visual information fusion to extract mid-term features. The neural network that extracts mid-term features is trained only through the results of downstream semantic segmentation and target detection. The supervision signal of the mid-term fusion is to maximize the accuracy of downstream tasks, but ignores the personalized needs of each collaborative participant, such as the task accuracy requirements of the robot for close-range objects are often higher than those for long-range objects.

[0009] Existing technologies usually run subsequent alignment and positioning processes directly based on the single-frame original perception data held by both parties of positioning. This will lead to a large waste of computing and communication resources in the time-series positioning scenario. Therefore, it is necessary to build a set of intelligent machine group communication and positioning methods that meet the objective communication conditions and reduce the waste of computing and communication resources. Summary of the invention

[0010] In response to the problems existing in the above-mentioned prior art, the present invention provides an integrated communication and positioning method for a group of intelligent machines. Communication-friendly mid-term features are extracted based on the actual communication capabilities and positioning requirements of individual intelligent machines, and the selection and scheduling of individual collaborative positioning objects are completed on the base station side. The individual intelligent machines estimate the relative position relationship between the machines based on the point cloud information of both parties, thereby realizing real-time operation under the constraints of objective communication conditions, and fusing cross-device features in the time and space dimensions to achieve efficient and high-precision continuous collaborative positioning, thereby improving the registration accuracy.

[0011] The present invention provides an integrated communication and positioning method for a group of intelligent machines. The application scenario includes multiple intelligent machines and multiple base stations. A mid-term feature extraction and interaction module and a time-series relative positioning module are deployed on each intelligent machine, and an optimal communication link construction module is deployed on each base station. The method of the present invention includes the following steps:

[0012] Step 1: The intelligent robot perceives through the lidar, processes the currently perceived radar point cloud data through the mid-term feature extraction and interaction module, constructs a query vector and broadcasts it, and extracts mid-term features for interaction. The mid-term feature extraction and interaction module of each intelligent robot constructs a multi-resolution feature map for the radar point cloud data it perceives, and then constructs a query vector for broadcasting. When robot A and robot B interact with each other through query vectors, the mid-term feature extraction and interaction module of robot A calculates the spatial importance of its own multi-resolution feature map to robot B based on the query vector of robot B, and then generates mid-term features with spatial importance from the multi-resolution feature map; robot A splices the mid-term features with the spatial importance features as interaction features and transmits them back to robot B.

[0013] Step 2: The base station first clusters the intelligent robots through the optimal communication link construction module, allocates the optimal communication links within the cluster, and then completes the communication links at the boundaries between clusters. Each intelligent robot initially tries to connect to all base stations in a broadcast manner, and then only retains the communication link with the highest signal strength, and classifies the intelligent robots connected to the same base station into one cluster; the optimal communication link construction module calculates the distance score between robots in the cluster based on signal strength and spatial importance, and then allocates collaborative positioning objects to the robots in the cluster according to the principle of maximizing the sum of the two-way distance scores between the intelligent robots in the cluster; for the robots at the cluster boundary, the distance score between them and the robots at the adjacent cluster boundary is calculated, and the communication link of the robots at the cluster boundary is optimized according to the distance score. The base station feeds back the collaborative positioning allocation result to the robot, and a direct communication link is established between the two collaboratively positioned robots.

[0014] Step 3: The intelligent robot receives the mid-term feature sequence sent by the collaborative positioning object. Based on the multi-resolution feature map sequence collected by itself and the mid-term feature sequence transmitted by the collaborative object, the robot performs point cloud registration through the time-series relative positioning module to obtain the relative position relationship between the robots. The time-series relative positioning module includes two feature extraction structures, a feature fusion device and a positioning output network; suppose robot A obtains a multi-resolution feature map sequence of itself at a continuous sampling time, and obtains a mid-term feature sequence of the corresponding continuous sampling time from robot B; the multi-resolution feature map sequence and the mid-term feature sequence are respectively input into a feature extraction structure to extract the aggregated spatiotemporal feature map; the feature fusion device fuses the aggregated spatiotemporal feature maps output by the two feature extraction structures, and inputs the fused features into the positioning output network; the positioning output network outputs the relative position relationship between the two robots at the current moment.

[0015] In the existing collaborative positioning technologies similar to the present invention, the use of time-series information is usually not considered. In order to ensure the collaborative effect, multi-sensor perception or external observer information must be introduced, and the stability of the output results is difficult to guarantee. The method of the present invention realizes efficient and fast collaborative positioning and communication in the multi-robot group interconnection scenario. Compared with the existing technology, it has the following advantages and positive effects:

[0016] (1) The method of the present invention takes into account the limitations of communication bandwidth constraints, designs a mid-term feature extraction and interaction module, and adaptively combines the positioning requirements of the robot to complete the sparse mid-term feature extraction and interaction of the original point cloud information, so as to control the communication scale required by the collaborative positioning algorithm. Existing relative positioning methods usually need to combine the perception data fed back by multiple sensors and devices operating in multiple environments. However, in actual application scenarios, due to the constraints of communication bandwidth and device computing units, too much perception data will cause the amount of data transmitted in real time to exceed the communication and computing carrying capacity of the system, thereby causing communication and computing bottlenecks. Therefore, the method of the present invention proposes a mid-term feature extraction network based on the actual task requirements of the collaborative party, and combines the local personalized requirements of the positioning task to complete the interaction and extraction of sparse feature maps between robots, so as to assist in achieving communication and computing-friendly collaborative positioning while ensuring task accuracy. Through efficient mid-term feature extraction and interaction strategies, the method of the present invention can flexibly adapt to various communication resource scenarios by simply adjusting the sparsity of the feature map during the interaction process.

[0017] (2) The method of the present invention realizes the construction of local optimal cooperative links and global clustering to cover the boundary conditions through the optimal communication link construction module based on signal strength and feature space importance, thereby avoiding the problem of communication and computing resource waste caused by distance factors to the greatest extent. Existing communication link construction methods usually construct resource scheduling optimization problems with the goal of optimizing system throughput and resource utilization. However, in the large-scale relative positioning scenario between robots facing the present invention, since relative positioning is only performed through perception data with overlapping conditions, most links do not meet this condition due to physical distance reasons, resulting in a large amount of communication and computing resources being wasted. Considering that the expected goal of the relative positioning method is to maximize the average effective amount of information obtained by the robots in the system about the relative position relationship with other robots in the communication topology, the construction method of the communication link in this scenario should be based on clustering based on the direct link of the communication base station to uniformly complete the intra-cluster communication relay, and design and execute the connection allocation and partial clustering algorithm based on the strategy of minimum relative distance between robots in the cluster and optimal connectivity between nodes under edge conditions between clusters. The method of the present invention avoids the huge uplink bandwidth pressure caused by full point-to-point communication in the system through the optimal communication link construction module, and at the same time avoids the problem of a large amount of communication and computing resource waste caused by the strong dependence of the collaborative positioning algorithm on data overlap through the methods of intra-cluster scheduling and inter-cluster completion.

[0018] (3) The method of the present invention improves the accuracy of positioning and registration by using the time-series information of historical point cloud information and historical prediction results through the time-series relative positioning module based on continuous communication frames. The existing relative positioning method does not consider the impact of time-series information on positioning, and mainly completes point cloud registration and subsequent positioning processes based on single-frame perception data. During the operation of the robot, there is a strong time-series correlation between the actual perception and relative positioning data, which will be reflected on both the input and output sides of the positioning algorithm. The scheme that completes the registration based only on single-frame perception information completely ignores the continuous frame information. From the perspective of information theory, in order to ensure the consistent performance of the positioning algorithm, additional computing or communication volume will need to be introduced to compensate for the loss of information. At the same time, with the assistance of the above-mentioned communication and mid-term feature strategies, the robot of the present invention can efficiently receive continuous mid-term feature frame sequences from multiple parties, creating conditions for the use of time-series information. The method of the present invention supplements and aggregates the information contained in the perception information in the spatiotemporal dimension, and at the same time performs time-attenuated output aggregation based on confidence weighting based on historical prediction results to achieve high-reliability relative positioning.

[0019] (4) The method of the present invention realizes real-time operation under the constraints of objective communication conditions, improves the utilization rate of information from a single perception data source, and at the same time, the output aggregation strategy based on confidence-time decay effectively ensures the stability of the collaborative algorithm output, expanding the application scenarios of collaborative positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of a scene where a group of intelligent machines are interconnected;

[0021] Figure 2 It is a framework diagram for realizing the communication and positioning integration method of the intelligent machine group of the present invention;

[0022] Figure 3 It is a realization framework diagram of the mid-term feature extraction and interaction module in the method of the present invention;

[0023] Figure 4 It is a framework diagram for realizing cross-robot joint information aggregation by the time-series relative positioning module in the method of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0025] like Figure 1As shown, in a scene of intelligent machine group interconnection, the intelligent robot obtains and processes the radar point cloud perception image, and the intelligent robot can communicate with the neighboring robot within its own communication range. The intelligent robots connected to the same base station are classified into a cluster, and the robots in the cluster and the base station form a subnet. The present invention provides an integrated communication and positioning method for intelligent machine groups. First, combined with the actual positioning needs of the recent intelligent machine group, medium-term features are extracted from the original perception information and the actual needs of the neighboring robots to reduce the communication needs. At the same time, the optimal communication cooperation cluster is constructed based on the optimal communication quality, and the scheduling and allocation of the connection and cooperation objects are further completed on the base station side. Finally, the main robot-neighboring robot feature pair is extracted based on the continuous communication frame sequence, and the point cloud registration is completed based on the local features of the two robots to obtain the relative position relationship between the two robots, and the relative positioning is completed.

[0026] like Figure 2 As shown, the integrated communication and positioning method for a group of intelligent machines in the embodiment of the present invention is mainly based on the computer to implement three functional modules: a mid-term feature extraction and interaction module based on relative positioning requirements, an optimal communication link construction module based on signal strength and feature space importance, and a timed relative positioning module based on continuous communication frames. A mid-term feature extraction and interaction module and a timed relative positioning module are deployed on each intelligent machine, and an optimal communication link construction module is deployed on each base station. The implementation of the embodiment of the present invention is described through the following three steps.

[0027] Step 1: Each intelligent machine extracts mid-term features from radar perception information through the mid-term feature extraction and interaction module.

[0028] Mark the current intelligent machine as the main robot. In order to control the resource consumption during the communication process, the present invention considers designing the main robot to complete relative positioning based on the feature map, and to interact with the feature map rather than the original point cloud data containing more redundant information during the communication process. On this basis, in order to improve the communication efficiency and ensure the accuracy of downstream positioning tasks, the feature extraction and interaction across robots and coordinate systems are completed through the query-response feature map interaction method. Therefore, the method of the present invention designs a mid-term feature extraction and interaction module based on the relative positioning requirements. Figure 2As shown in the figure, the mid-term feature extraction and interaction module of the main robot first constructs a multi-resolution feature map for the lidar point cloud data acquired by the local robot, and further constructs a low-dimensional query vector based on real-time needs for broadcasting. After exchanging the low-dimensional query vector with the neighboring robot, the main robot constructs a voxelized importance feature map with spatial importance information based on the query vector, and generates a mask matrix based on the importance feature map in the spatial dimension, and further constructs a sparse feature map based on the multi-resolution feature map with spatial importance weights, thereby realizing a communication-friendly mid-term feature extraction method that can be adjusted to various communication loads based on the scene.

[0029] Based on the importance of feature maps in the spatial dimension, the feature maps held by the neighboring robots are further cropped by masking operations, so that only the sparse feature maps corresponding to the channels that are more important to the main robot are retained, reducing the communication pressure in the subsequent collaborative process. Finally, the neighboring robots weight the sparse feature maps by importance, and transmit the sparse feature maps with implicit spatial importance back to the main robot to complete the extraction and interaction of mid-term features.

[0030] The implementation of the mid-term feature extraction and interaction module of the embodiment of the present invention is as follows Figure 3 As shown. In order to provide a neural network feature graph that meets the actual device communication requirements and is communication and computing friendly, it is necessary to build an adaptive feature extraction network in combination with downstream positioning tasks. In the embodiment of the present invention, the feature extraction network consists of a request encoder and a spatial local feature generator, and completes a round of data exchange process through a query-response communication interaction method.

[0031] In the collaborative localization process, the main robot first extracts the current point cloud into a multi-channel planar feature map from the BEV perspective. Wherein, H, W, and D represent the height, width, and depth of the feature map, respectively. The embodiment of the present invention uses the PointPillars pre-trained network to voxelize the point cloud into a nine-dimensional feature tensor representation: x, y, z, r, x c ,y c ,z c ,x p ,y p ; where x, t, a, r represent the three-dimensional spatial position and reflection intensity of a frame of point cloud, x c ,y c ,z c Represents the centroid position of all points in the Pillar to which the point cloud belongs, x p ,y p Represents the relative offset between the point and the centroid. The multi-resolution feature map of the point cloud is extracted through a deep convolutional network. Let the multi-resolution feature map extracted from the point cloud at time i be F i .

[0032] The embodiment of the present invention performs a further feature interaction process based on the feature map. The main robot sends the current feature map F i By requesting the encoder G q Compressed into a communication-friendly N-dimensional query vector Q i as follows:

[0033]

[0034] Among them G q is through θ q Parameterized trainable feature encoders.

[0035] In the communication interaction process, the query vector is used as a feature to reflect the external demand of the main robot. q Compressing the features to a low dimension can ensure that they are broadcast to each neighboring robot in the form of a request with low communication volume. The neighboring robot will then further reduce the dimension of its own feature map based on the request. Figure 3 As shown in Figure 2, suppose a neighbor robot first uses the PointPillars pre-trained network to extract the feature map F′ from its perceived point cloud data i , then for F′ i By requesting the encoder G q Compressed into a low-dimensional query vector Q′ i And broadcast.

[0036] During the communication interaction, the main robot converts the compressed low-dimensional query vector Q into i Broadcast to all co-location partners, i.e. neighboring robots, and receive query vector Q′ from neighboring robots i The main robot combines the feature map F of the local point cloud i The query vector Q′ with the neighboring robots i , and further judge the importance of the voxelized features in the spatial form of the feature map in the spatial dimension. Specifically, a single convolutional layer is used to define the voxelized importance generator Φ in the form of an object detection head. v , Q′ i With F i After splicing, calculate the spatial importance feature V i as follows:

[0037]

[0038] where Φ v is through θ v Parameterized Trainable Importance Generator, V i Represents the feature map F i The importance of each high-dimensional voxelized spatial feature to the neighboring robot. Generator Φv It is implemented with a plug-and-play target detection head structure, which can be replaced according to positioning requirements such as prediction time or accuracy. V is selected through the top-k strategy i The k values ​​with the largest weights are converted into a binary mask matrix M∈{0,1} H×W , set the values ​​of the positions corresponding to the k values ​​with the largest weights in the mask matrix M to 1, and the values ​​of the remaining positions to 0. The mask value 1 represents F i The features at the corresponding positions in M ​​are of high importance to the neighboring robots, and feature interaction is required. i The element-by-element dot product means to filter the original feature map based on the importance of spatial features, retaining only the features at the positions where the mask matrix of the feature map is 1, and discarding the features of the voxels where the mask matrix is ​​0, which do not participate in subsequent feature interactions, and obtain the sparse feature map F. i M Then F i M With V i Element-by-element point multiplication is performed to obtain the mid-term feature extraction and the mid-term feature C output by the interaction module through importance weighting. i .

[0039] Since the number of masks in M ​​is directly reflected in the dimension of the sparse feature map used in the final communication, and further directly affects the communication volume, based on the actual communication bandwidth conditions, the sparse feature map participating in the feature interaction can be changed by adjusting the k value to flexibly adapt to various communication environments. In addition, note that the neighbor robot V′ i The spatial importance information contained in C′ reflects i The importance of each voxel feature in C′ i And the spatial importance feature map V′ i After splicing, the features are transmitted back to the main robot as interactive features, completing a round of feature interaction process.

[0040] Step 2: The base station first clusters the intelligent robots through the optimal communication link construction module, allocates the optimal communication links within the cluster, and then completes the communication links at the boundaries between clusters.

[0041] In the scenario of large-scale robot group collaborative positioning, as the coverage of collaborative positioning expands, due to the limitations of objective physical conditions, point-to-point communication will obviously be limited by factors such as packet loss rate and communication bandwidth bottleneck and cannot be expanded in scale. In addition, the mid-term feature extraction and interaction module distributes feature maps in the form of broadcast. If a point-to-point communication solution is used, a large amount of additional uplink bandwidth will be wasted. Therefore, it is necessary to introduce the existence of communication base stations in the communication model, and the base stations complete the reception, processing and distribution of broadcast messages during the communication process. Considering that the ultimate goal of the collaborative system is to maximize the average amount of information of the effective relative positioning results obtained by the robots in the system, the method of the present invention aims to reduce invalid calculations and communications. First, the allocation of intra-cluster connections is completed based on the base station, and then the communication links that need to be established under the boundary conditions are further completed by clustering.

[0042] The optimal communication link construction module first initializes clustering according to the spatial importance of the feature graph and the signal strength between the base station. In the initial state, in order to build a communication link between the robot and the base station, the robot first broadcasts and tries to connect to all base stations, and selects and retains the communication link with the highest signal strength based on a greedy strategy, and cuts off the connection with the remaining base stations. At the same time, based on the actual computing resource load of the robot group, the maximum number of connections m that the robot can actually support can be obtained, and the base station further completes the selection of the robot's collaborative object and connection target.

[0043] It should be noted that in the process of base station-assisted robots selecting communication objects, it is difficult to directly judge the size of the distance between robots due to the lack of physical coordinates. The relative positioning algorithm based purely on perception information will obviously fail when the distance is too far due to the lack of overlapping relationship between perception information without additional information, and the connection established under such conditions will obviously waste a lot of communication and computing resources. In order to build a highly reliable and resource-efficient communication link, the m robots closest to each robot should be greedily selected, which requires a certain degree of prediction of the relative distance between robots. In the initialization stage, the main robot broadcasts the extraction of the query vector Q and the calculation of the voxelized importance V between robots, and obtains the voxelized importance held by all other connectable robots, marking the main robot as a, its neighbor robot as b, and V b is the spatial importance feature calculated by the neighbor robot b based on the query vector of the main robot a; calculate each V b Elements and D b , using D b represents the importance of the feature map of the main robot a to the neighbor robot b, and D b is considered as one of the factors reflecting the distance between the neighbor robot b and the main robot. This is because the voxelized importance generator Φ v Based on the downstream positioning task training, Vb It reflects the importance of the entire feature map of the neighbor robot to the main robot.

[0044] For two robots a and b in the same cluster, the signal strength between them and the base station is obtained as S a , S b , combined with the free space propagation model, the estimated distance between the two robots a and b can be obtained from the signal strength level as follows: However, this value will be misleading when the robots are distributed at the same distance from the base station but in different directions. Therefore, the present invention further weights the distance score R from the intelligent machine a to the machine b. ab as follows:

[0045]

[0046] Among them, α is an adjustable weight factor with a value range of (0,1) to control the weight of signal strength and bidirectional importance between devices in the distance scoring function, D ab Represents the importance of the feature map of robot b to robot a. Distance score R ab The larger it is, the better the positioning of robot b and robot a is.

[0047] According to the above method, the distance scores between all devices in the cluster can be obtained, and the base station establishes a two-way connection with the highest score for each robot. In a subnet of a base station, the optimal communication link construction module of the base station completes the connection scheduling process of the robots in the cluster according to the distance scores between devices, assigns collaborative positioning objects to the robots, and establishes communication interaction links. Assume that there are n robots in the base station subnet, and the maximum number of connections that a single robot can support is m. The base station maximizes the two-way scores between devices in the cluster and R sum The principle of assigning collaborative positioning objects to robots in the cluster is as follows:

[0048] R sum =∑(R ab +R ba ),a,b∈[1,n]

[0049] Among them, R ab is the distance score from intelligent machine a to machine b, R ba is the distance score from intelligent machine b to machine a.

[0050] For a subnet of a base station, each robot needs to set up m collaborative positioning objects. The base station will feed back the collaborative positioning allocation results to the robot, and a direct communication link will be established between the two collaboratively positioned robots.

[0051] When a robot leaves the current subnet or the difference in signal strength between two robots with assigned communication links When it is less than the set threshold, the base station needs to recalculate the distance scores between devices for the affected robots and reallocate the co-positioning objects and communication links.

[0052] After completing the connection allocation of the intelligent robots in the cluster, the two robots located at the boundary of the two communication clusters are assigned to different clusters despite their relatively close physical distance, and communication cannot be established between the two. In view of this situation, the optimal communication link construction module of the present invention realizes the completion of the inter-cluster boundary link by cross-base station communication, inter-cluster collaboration and clustering to a certain extent. Considering the coverage range of the communication base station, the robot closer to the cluster center obviously does not need to communicate across clusters. Therefore, the embodiment of the present invention regards the robot whose signal strength S and all m distance scores R within the communication base station cluster are lower than the preset threshold as being located at the cluster boundary. For the robot a located at the boundary of the base station, the robot b located at the boundary among the adjacent base stations of the base station is selected, and the distance scores between the two are calculated again by cross-cluster method, and whether to replace the communication link that has been constructed between robot a and robot b is selected based on the calculated distance score. For example, if the calculated distance score is higher than the distance score between robot a and a certain robot, a communication link can be constructed between robot a and robot b to replace the communication link with a low distance score that already exists between robot a and robot b. The distance score R from robot a to robot b is calculated across clusters ab The calculation is done in the same way as the distance scores between robots within a cluster, but the difference is that the signal strength is provided by the base station to which the robot is connected.

[0053] Through the above method, the optimal communication link construction module realizes low-latency and long-distance networking between the intelligent machine group and the communication base station, which can ensure the real-time and reliability of collaborative positioning and reduce the consumption of communication and computing resources within the system.

[0054] Step 3: According to the interactive communication characteristics of the constructed communication link, each intelligent machine receives a continuous sequence of communication frames from its neighboring machine and determines the relative position between the two machines through point cloud registration using the timed relative positioning module.

[0055] Through the above steps, the robot can efficiently communicate with the best neighbor robot within the scope allowed by the computing and communication load to exchange their respective real-time point cloud feature maps and their importance features in the spatial dimension. After that, in order to achieve high-precision relative positioning, it is necessary to pay attention to the continuity characteristics shared by the sparse feature map data and the robot position information in the temporal relationship. The existing point cloud registration algorithm usually targets a single point cloud pair at a specific moment. Although this type of method can ensure that the registration distance error reaches a relatively optimal solution in a statistical sense, when it is directly applied to real-time positioning, due to its neglect of temporal continuity, when running on a continuous point cloud sequence, there will often be a huge inter-frame error and the confidence of the result cannot be guaranteed at all. It is difficult to guarantee its availability in the scenario of continuous registration. In order to solve the above problems, it is necessary to efficiently perform joint information aggregation on the feature maps distributed in different feature spaces with implicit temporal features across robots, so as to align the features from different sources and at different times to the same feature space.

[0056] The main robot receives the interactive features from the neighboring robots, extracts the mid-term features calculated by the neighboring robots, and uses the multi-resolution feature maps of the local robot at continuous moments and the mid-term features of the neighboring robots at continuous moments as the input of the temporal relative positioning module. The temporal relative positioning module targets the continuous feature maps of the main robot and the neighboring robots respectively. On the one hand, it extracts intra-frame static attention from them through the self-attention mechanism, that is, it divides the feature map sequence into a sequence of voxelized feature blocks that are continuous in the time dimension and independent in the spatial dimension, and extracts spatiotemporal features from the local time to the global space perspective. On the other hand, it extracts inter-frame dynamic attention from them through the self-attention mechanism based on the feature map divided in the time dimension, that is, it extracts spatiotemporal features from the global perspective of the feature map;

[0057] Then, the heterogeneous features from different robots are further fused through the cross-attention mechanism. Finally, the features after aggregating multi-source temporal information are used for positioning and adaptive confidence extraction, and a weighted output aggregation operation is performed based on the historical positioning and confidence results to complete the prediction process at one time step.

[0058] like Figure 4 As shown, the temporal relative positioning module of the present invention is implemented based on the improved Transformer structure, including a first feature extraction structure, a second feature extraction structure, a feature fusion device and a positioning output network. The first feature extraction structure and the second feature extraction structure both include a one-dimensional (1-D) convolution module and N encoding modules. Each encoding module is an improvement of the encoding module based on the Transformer structure. Two multi-head self-attention modules are set at the input, which are used to extract global spatiotemporal features and local spatiotemporal features respectively. Suppose that the multi-resolution feature map sequence F at a continuous sampling time obtained by the main robot at the current time t is i(i=t-T+1,…,t), obtain the interaction feature C of a neighbor at the corresponding continuous sampling time i (i=t-T+1,…,t); Assume that the continuous sampling time length is T, and the T value can be adjusted. i Input the first feature extraction structure, and extract the aggregated spatiotemporal feature map of the main robot from it. i The first feature extraction structure is input to extract the aggregated spatiotemporal feature map of the neighboring robot. The feature fusion device contains N decoding modules, which are implemented using the decoding module of the Transformer structure. The aggregated spatiotemporal feature maps of the main robot and the neighboring robot are fused, and then the fused features are input into the positioning output network. The positioning output network outputs the relative position relationship of the two robots at the current time t through the fully connected positioning network, including the predicted position coordinates and rotation angle. The number of superpositions of the encoding module or the decoding module N is a positive integer and can be adjusted according to the required accuracy and time requirements.

[0059] In the first feature extraction structure, the following operations are performed: (1) F i (i=t-T+1,…,t) is integrated into the spatiotemporal feature map S f , and is divided into voxelized feature block sequences s in the spatial dimension (f,j) , j = 1, 2, ..., H*W, perform 1-D convolution operation on the voxelized feature block sequence, aggregate the local spatial information of the block sequence in the depth dimension through 1-D convolution, and then input the aggregated local spatial feature sequence into a multi-head self-attention module, and calculate the relationship between the feature blocks through the multi-head temporal attention at the time level, that is, obtain the local spatiotemporal features that integrate the intra-frame attention; (2) F i (i=t-T+1,…,t) inputs another multi-head self-attention module to extract the inter-frame attention of the feature map and obtain the global spatiotemporal features from a global perspective; (3) i The local and global spatiotemporal features extracted from sequences and F i The sequence is processed by residual connection and then input into the feedforward network. The input and output of the feedforward network are processed by residual to obtain the aggregated spatiotemporal feature map H extracted from the information collected by the main robot. ego Similarly, in the second feature extraction structure, the continuous feature map C i (i=t-T+1,…,t) is integrated into the spatiotemporal feature map S c , voxelized into a feature block sequence s in the spatial dimension (c,j) , j = 1, 2, ..., H * W, the local spatial features in the depth dimension are aggregated by 1-D convolution operation, and then a multi-head self-attention module is input to extract the local spatiotemporal features of the intra-frame attention. i(i=t-T+1,…,t) inputs a multi-head self-attention module to extract the global spatiotemporal features of the fused inter-frame attention. Then C i The extracted global spatiotemporal features and local spatiotemporal features are fused into the aggregated spatiotemporal feature map H of the neighboring robots. neighbor .

[0060] In the embodiment of the present invention, S f or S c Divided feature blocks The sequence is input into a 1-D convolution layer with a convolution kernel size of k*D for local feature aggregation, and the local features are as follows:

[0061]

[0062] Among them, h,w,t is the local feature extracted by the convolution kernel at the feature block s at the position (h, w, t), s(h, w, d, i) is the feature with a depth of d at the position (h, w) at time i, and K d ∈R k*D represents the convolution kernel used in 1-D convolution, and k represents the size of the convolution kernel in the depth dimension. In this embodiment of the present invention, the H×W×D×T dimensional data composed of the feature block sequence is converted into H×W×T dimensional data through 1-D convolution. Each feature block after aggregation The local temporal information carried in consecutive frames is aggregated into the feature map to enhance the model's ability to capture the temporal changes of local features.

[0063] Then, o is spliced ​​into a serialized feature map and input into the multi-head self-attention mechanism to further aggregate the mutual relationship between the local feature spaces in the sequence, and the local spatiotemporal features associated with the local time series and the global space are obtained. The global spatiotemporal features from the global space perspective are directly extracted from the feature map sequence, and the local spatiotemporal features and the global spatiotemporal features are spliced ​​and fused to obtain the aggregated spatiotemporal feature map from local to global. Let the aggregated spatiotemporal feature maps extracted from the main robot be H ego , the aggregated spatiotemporal feature map extracted from the neighboring robots is H neighbor . Since the neighbor robot returns C i The voxelized importance V i , H neighbor It will naturally carry the voxel spatial importance information from the perspective of the main robot, which will facilitate the subsequent fusion of the main robot and neighbor robot feature maps. ego and H neighbor The data dimensions are all H×W×T.

[0064] The feature fusion device obtains the temporal feature graph H of the main robot and the neighboring robots ego With Hneighbor After that, the cross-robot feature fusion is performed through the cross-attention mechanism to obtain the fusion feature A at the current time t t as follows:

[0065]

[0066] Among them, CrossAttn Wx is a criss-cross attention layer parameterized by Wx.

[0067] Due to H neighbor The voxel space has the natural characteristics of importance, which can effectively align the cross-robot features to the same spatial dimension. At the same time, since the input feature map sequence already carries temporal features, H is fused through cross attention. ego With H neighbor The output obtained is A t It will be possible to align the temporal features across robots and coordinate systems into the same feature space in both time and space dimensions. Output A of the feature fusion unit t The data dimension is H×W.

[0068] Finally, the positioning output network obtains the predicted value of the relative position relationship between the point cloud pairs input at the current time t through the dual-head output Predicted value with rotation angle The output matrix P t And the confidence c of the current prediction result after softmax normalization t In order to solve the problem of extremely large inter-frame errors and unreliable confidence in results, the present invention implements a time-decayed output aggregation solution based on confidence weighting for a fixed time window T. Specifically, for the current moment t, combined with the historical output matrix P in the time window T i and confidence c i , i = t-T+1,…,t-1, before using the output of the model, calculate O t The final output is as follows:

[0069]

[0070] where α i It is a combination of confidence and decay factor weight obtained by time step, defined as:

[0071]

[0072] Where γ∈[0,1] is a predefined attenuation factor that reduces the impact of historical time steps on the output results as the time step increases.

[0073] The timing relative positioning module of the present invention outputs the matrix P at the current time t. t , confidence c t And the final output result O t According to O t Determine the relative position relationship between machines at the current time t, and output the matrix P t and confidence c t It will be used as historical data for prediction at subsequent times.

[0074] The temporal relative positioning module of the present invention is based on continuous perception data and continuous communication frames, and completes the spatiotemporal feature aggregation of serialized single-source feature maps from a global perspective and a local time-global space dimension to make full use of the implicit temporal correlation in continuous frames. After that, the sparse feature map sequences from neighboring robots are aggregated through a cross-attention mechanism to complete the fusion of multi-source, cross-device temporal feature maps. In order to further improve the reliability of the prediction output, based on the confidence and time step of the historical prediction output, a time-decayed weighted output aggregation strategy is performed in the form of a sliding window.

[0075] The method of the present invention is verified and compared with the perception data collected by standard robot equipment. In the real-time operation process, the average bandwidth occupancy caused by the operation of transmitting continuous raw point cloud perception information between devices is about 120Mbps, while under the medium-term feature interaction with the preset top-k as half the size of the feature map, the average bandwidth occupancy of feature sharing between devices is reduced to 4.6Mbps, which provides the possibility for large-scale robot cluster collaboration. In addition, in the actual verification process, after training with the same time step data, that is, a total of 300,000 point cloud perception data pairs, the existing optimal point cloud registration scheme BUFFER has an average positioning error of 5.73cm and a single prediction time of 0.2 seconds in the verification set. The average positioning error of the method of the present invention is 2.67cm, and the single prediction time including the perception data interaction process is 0.37 seconds. It is proved that the present invention can effectively avoid the huge communication overhead caused by the transmission of raw perception data under an acceptable amount of additional calculation, and at the same time, efficiently utilize the perception data to complete high-precision collaborative positioning.

[0076] In summary, it can be seen that by adopting the method of the present invention, the amount of interactive feature data can be adjusted according to the objective communication conditions to meet the requirements of real-time operation, and multi-source time series data across devices can be integrated. The time series relationship of the relative positioning data is taken into account, which can effectively improve the accuracy of relative positioning and the information utilization rate, and can improve the continuity and credibility of the relative positioning algorithm in terms of time series.

[0077] Except for the technical features described in the specification, all of them are known technologies to those skilled in the art. The present invention omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.

Claims

1. A method for integrating communication and positioning of a group of intelligent machines, applied to the interconnection scenario of a group of intelligent machines, characterized in that: The method comprises the following steps: Step 1: The intelligent robot perceives through the lidar, and uses the mid-term feature extraction and interaction module to construct and broadcast the query vector based on the currently perceived radar point cloud data, and extract mid-term features for interaction; The mid-term feature extraction and interaction module of each intelligent robot constructs a multi-resolution feature map for the radar point cloud data it perceives, and then reconstructs a query vector for broadcasting; when robot A and robot B exchange query vectors, the mid-term feature extraction and interaction module of robot A calculates the spatial importance of its own multi-resolution feature map to robot B based on the query vector of robot B, and then generates mid-term features with spatial importance; robot A splices the mid-term features with the spatial importance features as interaction features and transmits them back to robot B; Step 2: The base station first allocates the optimal communication links to the intelligent robots in the cluster through the optimal communication link construction module, and then completes the communication links between the intelligent robots at the boundaries between clusters; Each intelligent robot initially broadcasts and tries to connect to all base stations, then only retains the communication link with the highest signal strength, and groups the intelligent robots connected to the same base station into a cluster; the optimal communication link construction module calculates the distance scores between robots in the cluster based on signal strength and spatial importance, and then allocates collaborative positioning objects to the robots in the cluster according to the principle of maximizing the sum of the two-way distance scores between the intelligent robots in the cluster; for robots located at the edge of the cluster, the distance scores between them and robots located at the edge of the adjacent cluster are calculated, and the communication links of robots located at the edge of the cluster are optimized according to the distance scores; the base station feeds back the allocated objects of collaborative positioning to the robot, and establishes a direct communication link between the two collaboratively positioned robots; Step 3: The robots that establish the communication link communicate with each other. Based on the multi-resolution feature map sequence collected by the robot and the mid-term feature sequence of the continuous communication frame transmitted by the collaborative positioning object, the robot uses the time-series relative positioning module to perform point cloud registration to obtain the relative position relationship between the robots. The temporal relative positioning module includes two feature extraction structures, a feature fusion device and a positioning output network; it is assumed that robot A obtains a multi-resolution feature map sequence of its own continuous sampling time at the current moment, and obtains a medium-term feature sequence corresponding to the continuous sampling time from robot B; the multi-resolution feature map sequence and the medium-term feature sequence are respectively input into a feature extraction structure to extract an aggregated spatiotemporal feature map; the feature fusion device fuses the aggregated spatiotemporal feature maps output by the two feature extraction structures, and inputs the fused features into the positioning output network; the positioning output network outputs the relative position relationship between the two robots at the current moment.

2. The method according to claim 1, characterized in that In step 1, the mid-term feature extraction and interaction module of robot A implements the functions of: firstly using the PointPillars pre-trained network to represent the radar point cloud data as a nine-dimensional feature tensor, extracting the multi-resolution feature map F of the point cloud at time i i ; Then use the request encoder G q The feature map F i Compressed into a communication-friendly query vector Q i And broadcast it; after receiving the query vector Q′ sent by the neighbor robot B i After that, we use the voxelized importance generator Φ v Calculate the feature map F i The importance of each voxelized spatial feature in , let the generated spatial importance feature map V i ; Use the top-k strategy to select V i The k values ​​with the largest weights in the matrix are set to 1 and the other positions are set to 0 to obtain a binary mask matrix M. i Element-wise dot product, keeping only F i Corresponding to the features at the position where the value is 1 in the mask matrix, the features at the position where the value is 0 in the mask matrix are discarded to obtain the feature map F i M , and then F i M With V i Element-by-element point multiplication is performed to obtain a sparse matrix graph weighted by spatial importance, i.e., the mid-term feature C i ; Robot A will mid-term feature C i And the spatial importance feature map V i Splicing as a feature for interaction with robot B.

3. The method according to claim 1 or 2, characterized in that: In step 1, the spatial importance of the multi-resolution feature map of robot A to robot B is calculated as follows: The query vector Q′ of robot B i and the multi-resolution feature map F of robot A i After concatenation, the voxelized importance generator is used to calculate F i The spatial importance of V to robot B i ; Among them, the voxelized importance generator Φ v It is implemented as an object detection head using a single convolutional layer.

4. The method according to claim 2, characterized in that: In the step 1, the k value is adjusted according to the actual communication bandwidth to change the sparse feature map involved in the feature interaction.

5. The method according to claim 1, characterized in that In step 2, the optimal communication link construction module calculates the distance scores between robots in the cluster as follows: Assume that the signal strength between the two robots a and b in the cluster and the base station is S a , S b , the spatial importance of the multi-resolution feature map of robot b to robot a is V ab , calculate V ab Elements and D ab , then the distance score R from robot a to robot b ab as follows: Among them, α is the weight factor, and its value range is (0,1).

6. The method according to claim 1 or 5, characterized in that: In step 2, the optimal communication link construction module calculates the distance scores between all robots in the cluster, and maximizes the bidirectional scores and R between devices in the cluster according to the maximum number of connections m that can be supported by a single robot. sum Assign collaborative positioning objects to the robots in the cluster, and assign m collaborative positioning objects to each robot; where R sum =∑(R ab +R ba ),a,b∈[1,n]; n is the total number of robots in the cluster, R ab is the distance score from robot a to robot b, R ba is the distance score from robot b to a; When a robot leaves the current cluster or the signal strength difference between the two robots in the assigned communication link and the base station is less than the set threshold, the optimal communication link construction module recalculates the distance scores between the affected robot devices in the cluster and reallocates the collaborative positioning objects.

7. The method according to claim 1 or 5, characterized in that: In step 2, if the signal strength S of the robot's connection with the base station and the distance score to the assigned collaborative object in the cluster are both lower than the preset threshold, the robot is considered to be located at the cluster boundary; for two robots located at the boundary of adjacent base station clusters, the distance score between the two robots is calculated based on the signal strength and spatial importance, and the collaborative positioning objects of the two robots are updated according to the distance score.

8. The method according to claim 1, characterized in that In step 3, both feature extraction structures include a one-dimensional convolution module and N encoding modules. The encoding module is obtained by replacing the multi-head self-attention module of the encoding module of the Transformer structure with two multi-head self-attention modules. Robot A obtains its own multi-resolution feature map sequence F at time t. i (i=t-T+1,…,t) and the mid-term feature sequence C of robot B i (i=t-T+1,…,t); on the one hand, the multi-resolution feature map sequence of robot A or the mid-term feature sequence of robot B is integrated and divided into a voxelized feature block sequence in the spatial dimension, and the one-dimensional convolution module convolves the voxelized feature block sequence, aggregates the local spatial feature sequence in the depth dimension and inputs it into a multi-head self-attention module, calculates the mutual relationship between the voxelized feature blocks, and obtains the local spatiotemporal features that integrate the intra-frame attention; on the other hand, the multi-resolution feature map sequence of robot A or the mid-term feature sequence of robot B is input into another multi-head self-attention module to extract the global spatiotemporal features that integrate the inter-frame attention; In the encoding module, local spatiotemporal features and global spatiotemporal features are aggregated. Suppose the aggregated spatiotemporal feature map H is obtained from the multi-resolution feature map sequence of robot A. ego , the aggregated spatiotemporal feature map H is obtained from the mid-term feature sequence of robot B neighbor ; Where T is the length of continuous sampling time and N is a positive integer.

9. The method according to claim 1 or 8, characterized in that: In step 3, the feature fusion device is implemented using N Transformer structure decoding modules, where N is a positive integer; the feature fusion device obtains the aggregated spatiotemporal feature map H extracted from the multi-resolution feature map sequence of robot A at the current time t ego And the aggregated spatiotemporal feature map H extracted from the current mid-term feature sequence of robot B neighbor , use the cross-attention mechanism to fuse the features across robots and obtain the fused feature A at the current time t t as follows: A t =CrossAttn Wx (H ego ,H neighbor ,H neighbor ) Among them, CrossAttn Wx is a criss-cross attention layer parameterized by Wx.

10. The method according to claim 1 or 8, characterized in that: In step 3, the positioning output network predicts the relative position relationship between the two robots A and B based on the fusion features at the current time t. With rotation angle And the confidence of the prediction result c t ; Let the current relative position relationship and rotation angle form the output matrix P t , combined with the historical output matrix P within the time window T i and confidence c i , i = t-T+1,…,t-1; calculate the final output matrix O at the current time t t : Among them, α i is the confidence-weighted time decay factor at time i, calculated as: Where γ is a predefined attenuation factor, γ∈[0,1].

Images