A method for collaborative beyond-visual-range perception and group intelligent decision-making of heterogeneous traffic entities

Abstract

The present invention provides a method for collaborative beyond-visual-range perception and group intelligent decision-making of heterogeneous traffic entities. The method includes: the main vehicle extracts cross-modal multi-scale features of on-board camera images and radar point cloud data, implements deep fusion perception calculation based on transformer cross-attention mechanism, and obtains deep fusion perception data; heterogeneous traffic entities adaptively transmit original data or deep fusion perception data to the main vehicle according to collaborative rules; from the perspective of the autonomous driving main vehicle, the multi-source data of the main vehicle and other collaborative heterogeneous traffic entities are integrated to realize beyond-visual-range collaborative perception among multiple traffic entities, and the main vehicle adopts a decision-making algorithm based on group intelligence according to the collaborative perception results of the heterogeneous traffic entities to obtain the optimal driving decision result of the main vehicle. The method of the present invention provides a high-quality data foundation for path search optimization based on the beyond-visual-range collaborative perception method of heterogeneous traffic entities of vehicles and roads, promotes the formation of a group intelligent decision-making theoretical system, and assists the research and development and application of high-level autonomous driving vehicles.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traffic vehicle-road collaborative technology, and in particular to a method for collaborative beyond-visual-range perception and group intelligent decision-making of heterogeneous traffic entities. Background Art

[0002] The development of intelligent transportation systems is the main way and means to alleviate current traffic problems. How to use advanced technologies such as machine learning, computer vision and group control to improve the intelligence level of vehicles is an urgent problem to be solved. Perception, as the data input for vehicles to obtain location information and traffic environment conditions, is the key to affecting the vehicle's autonomous decision-making and control capabilities and operational safety and efficiency. Inaccurate perception results and incomplete perception range are the two main reasons for traffic accidents and poor vehicle decision-making.

[0003] The use of a single sensor for perception is insufficient in terms of reliability and accuracy. Radar point clouds and visual images are of great importance in the field of intelligent traffic perception due to their data characteristics. How to overcome the excessive reliance of traditional data association on the spatial transformation matrix and realize the deep fusion perception of heterogeneous data is an issue that cannot be ignored. In addition, the perception distance of radar and camera sensors is limited. It is difficult to meet the needs of fast-moving vehicles for comprehensive perception of the traffic environment situation with only on-board sensors. The emergence of C-V2X networks provides an opportunity for breakthroughs in vehicle-to-vehicle and vehicle-to-road beyond-visual-range collaborative perception technology. However, collaborative perception enhancement is different from vehicle-to-vehicle autonomous perception. For group intelligent decision-making and optimization tasks, vehicles need to balance communication transmission load, perception computing performance, and multi-agent collaborative effects to provide more comprehensive and accurate traffic situation perception data for decision-making optimization algorithms through efficient vehicle-to-road collaborative beyond-visual-range perception.

[0004] At present, there is a lack of beyond-line-of-sight perception methods that can deeply integrate and coordinate cross-modal and multi-source data of heterogeneous traffic entities and comprehensively consider communication and computing capabilities in order to solve the problem of group intelligent decision-making in a networked information interaction environment. Summary of the invention

[0005] The present invention provides a method for collaborative beyond-visual-range perception and group intelligent decision-making of heterogeneous traffic entities to solve the problems of insufficient vehicle-mounted autonomous perception capability, poor effect and limited range in the group intelligent decision-making process.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme.

[0007] A method for collaborative beyond-visual-range perception and group intelligent decision-making of heterogeneous traffic entities, comprising:

[0008] The main vehicle extracts cross-modal and multi-scale features of the on-board camera image and radar point cloud data, implements deep fusion perception calculation based on the transformer cross-attention mechanism, and obtains deep fusion perception data;

[0009] Based on the constraints of the communication range of the main vehicle, the coordinated heterogeneous traffic subjects are determined, and the heterogeneous traffic subjects adaptively transmit raw data or deeply fused perception data to the main vehicle according to the coordination rules;

[0010] From the perspective of the autonomous driving vehicle, the multi-source data of the vehicle and other coordinated heterogeneous traffic entities are integrated to achieve beyond-visual-range coordinated perception among multiple traffic entities, and obtain coordinated perception results of heterogeneous traffic entities;

[0011] The main vehicle adopts a decision-making algorithm based on swarm intelligence according to the collaborative perception result of the heterogeneous traffic entities to obtain the optimal driving decision result of the main vehicle.

[0012] Preferably, the main vehicle extracts cross-modal multi-scale features of the vehicle-mounted camera image and the radar point cloud data, implements deep fusion perception calculation based on the transformer cross-attention mechanism, and obtains deep fusion perception data, including:

[0013] The main vehicle extracts keys and values ​​from image data rich in semantic texture information through a deep fusion method based on a transformer cross-attention mechanism using on-board sensors, generates queries using point cloud data with sparse and disordered three-dimensional spatial information, and uses high-resolution visual images to guide low-resolution radar point clouds to complete target detection environment perception tasks, thereby achieving deep fusion perception calculations and obtaining deep fusion perception data.

[0014] The transformer cross-attention mechanism network includes a backbone network, a multi-scale feature extraction layer, a fusion network and a detection head. The backbone network uses ResNet50 to mainly extract image features; the multi-scale feature extraction layer mainly selects the effective features extracted by the backbone network to fully obtain data information of different scales; the fusion network realizes the deep fusion of radar data features according to the encoding and decoding structure; the detection head outputs the results of perception tasks such as prediction boxes and category confidence.

[0015] The calculation formula based on the transformer cross attention mechanism is as follows:

[0016] Q=W Q X L ,K=W K X C ,V=W V X C

[0017]

[0018]

[0019] Where Q, K, and V are query, key, and value matrices respectively, and W Q , W K , W V are their corresponding weight parameters, X L , X C are radar and camera sensor data inputs respectively, α is the normalized dot product attention calculation, CressAttention is the cross attention calculation, and d k is the dimension of K vector;

[0020] The position encoding of the Transformer cross-attention mechanism network uses a two-dimensional rotation encoding as shown below:

[0021]

[0022] In the formula, θ i =10000 -2i / d , and d is the dimension of the feature vector.

[0023] Preferably, the determining of the coordinated heterogeneous traffic subject based on the constraint of the communication range of the main vehicle, and the heterogeneous traffic subject adaptively transmitting the original data or the deeply fused perception data to the main vehicle according to the coordination rules, includes:

[0024] Based on the communication range constraints of the autonomous driving main vehicle, determine the coordinated heterogeneous traffic entities such as vehicles and roads, where the heterogeneous traffic entities include adjacent vehicles and intelligent roadside equipment, and the roadside equipment includes digital traffic infrastructure with autonomous perception, computing, and networked communication capabilities;

[0025] Determine whether it is the first collaboration between the autonomous driving main vehicle and the heterogeneous traffic subject. If so, the heterogeneous traffic subject transmits the original sensor data collected by its sensors to the cloud computing server; otherwise, the heterogeneous traffic subject transmits the deep fusion perception data to the main vehicle.

[0026] Preferably, in the perspective of the autonomous driving main vehicle, the multi-source data of the main vehicle and other collaborative heterogeneous traffic entities are integrated to realize the beyond-visual-range collaborative perception among multiple traffic entities, and obtain the collaborative perception results of the heterogeneous traffic entities, including:

[0027] If the number of coordinated heterogeneous traffic entities is greater than or equal to 3, the cloud computing center provides computing resource support to assist the autonomous driving vehicle in completing beyond-visual-range collaborative perception calculations. If the number of coordinated heterogeneous traffic entities is less than 3, the autonomous driving vehicle uses the on-board terminal and the edge computing center to complete the perception solution.

[0028] From the perspective of the autonomous driving vehicle, the multi-source data of the autonomous driving vehicle and other coordinated heterogeneous traffic entities are integrated to achieve beyond-visual-range collaborative perception among multiple traffic entities. The multi-source data is temporally and spatially aligned, and the satellite positioning clock source is used to provide a time reference. The time stamp and interpolation and extrapolation method of the multi-source data are used to achieve time alignment. The spatial alignment is achieved based on the relationship between the local coordinate system of the autonomous driving vehicle, the local coordinate systems of other heterogeneous traffic entities, and the global coordinate system. The specific process is as follows:

[0029] G V =[R V T V ]L V

[0030] G O =[R O T O ]L O

[0031] L VO =[R V T V ] -1 [R O T O ]L O

[0032] In the formula, G V is the perception data of the main vehicle in the global coordinate system, L V G is the perception data of the main vehicle in the local coordinate system of the main vehicle, O is the perception data of other coordinated heterogeneous traffic entities in the global coordinate system, L O is the self-perception data of other coordinated heterogeneous traffic entities in the local coordinate system, R V , R O and T V 、T O are the corresponding rotation and translation matrices, L VO The perception data of other coordinated heterogeneous traffic entities in the local coordinate system of the main vehicle;

[0033] The collaborative perception results of heterogeneous traffic entities of vehicles and roads are determined by combining the intersection-and-joint ratio and confidence. When the intersection-and-joint ratio of the perception results between heterogeneous multiple traffic entities is greater than the threshold N, it is considered that the perceived targets are the same, and the results with higher confidence are retained; otherwise, they are all retained. Based on the above processing process, the main vehicle obtains the collaborative perception results of heterogeneous traffic entities including the position, speed, direction of obstacles and other road participants in the traffic environment, as well as traffic sign information.

[0034] Preferably, the host vehicle uses a decision-making algorithm based on swarm intelligence cooperative evolution to obtain the optimal driving decision result of the host vehicle according to the collaborative perception result of the heterogeneous traffic entities, including:

[0035] According to the beyond-visual-range collaborative perception results of heterogeneous traffic entities and the path nodes involved, the driving environment is uniformly characterized through a decision-making algorithm based on the cooperative evolution of swarm intelligence, and the collaborative heterogeneous traffic entities are regarded as ant colonies. Each heterogeneous traffic entity simulates the pheromone release and volatility characteristics of ants during the driving process. The speed and position of individuals in the heterogeneous traffic group are constantly changing, and the road and path weight parameters of the heterogeneous traffic entities are iteratively updated. The optimal path information is obtained by using swarm intelligence, and finally the optimal driving decision result of the main vehicle is output. The optimal driving decision result includes the acceleration, deceleration and steering decisions of the main vehicle at the current moment, as well as the optimal driving path to the destination.

[0036] Preferably, the host vehicle uses a decision algorithm based on swarm intelligence cooperative evolution to obtain the optimal driving decision result of the host vehicle according to the collaborative perception result of the heterogeneous traffic entities, and further includes:

[0037] Determine the driving purpose of the autonomous driving vehicle, obtain the potential feasible solutions of all optimal paths for the vehicle to reach the destination, define the necessity and optimality of each path as the path weight, the larger the path weight, the more likely the vehicle should choose this path, and initialize the path weight parameters;

[0038] The transfer probability of heterogeneous traffic entities under the current traffic environment is iteratively calculated according to the path weight parameter, and the possibility of heterogeneous traffic entities from the current position to the next path node is determined as follows:

[0039]

[0040] Where t is the current time, P ij is the transition probability of a heterogeneous traffic subject from the current location i to the next path node j via the path, ω ij (t) is the path weight parameter, is the safety risk factor of path ij obtained based on collaborative sensing information, is the sum of the calculation results of all possible path nodes, α is the path weight importance factor, and β is the collaborative perception information importance factor;

[0041] The principles for iterative updating of path weight parameters are as follows:

[0042]

[0043] In the formula, ρ is the weight decay parameter, is the weight of the mth coordinated vehicle’s contribution to path ij, Δω ijis the sum of the weight increments of path ij at time t, where:

[0044]

[0045] Where k is the kth iteration, L k,m is the current path distance of the mth vehicle, L min is the optimal path distance, L max is the worst path distance, ε is the threshold constant;

[0046] Determine whether the maximum number of iterations has been reached. If not, continue the above-mentioned iterative calculation process; otherwise, output the optimal driving decision result of the host vehicle based on swarm intelligence.

[0047] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the present invention can solve the problem of insufficient autonomous perception ability of vehicles in group intelligent decision-making, provide a high-quality data basis for path search optimization based on the beyond-visual-range collaborative perception method of heterogeneous traffic entities between vehicles and roads, promote the formation of a theoretical system of group intelligent decision-making, and assist in the research and development and application of high-level autonomous driving vehicles.

[0048] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0050] Figure 1 A schematic diagram of the implementation principle of a method for collaborative beyond-visual-range perception and group intelligent decision-making of heterogeneous traffic entities provided in an embodiment of the present invention;

[0051] Figure 2 A processing flow chart of a method for collaborative beyond-visual-range perception and group intelligent decision-making of heterogeneous traffic entities is provided for an embodiment of the present invention;

[0052] Figure 3 A C-V2X-supported vehicle-road heterogeneous traffic subject collaborative perception map is provided in an embodiment of the present invention.

[0053] Figure 4 A beyond-visual-range collaborative perception and group intelligent decision-making scenario diagram provided by an embodiment of the present invention.

[0054] Figure 5A diagram of the collaborative beyond-visual-range perception effect of heterogeneous traffic entities provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0056] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0058] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0059] The embodiment of the present invention provides a method for collaborative over-the-horizon perception and group intelligent decision-making of heterogeneous traffic entities, which realizes more comprehensive and accurate perception and recognition of traffic environment situation, and supports vehicles to make better decisions such as path search. The schematic diagram of the implementation principle of this method is shown in the figure Figure 1 As shown, including:

[0060] The situation deep fusion perception method based on the cross-attention mechanism comprehensively utilizes the heterogeneous data information of different sensors such as visual cameras and lidars, fully mines the color semantic information of visual images and the three-dimensional spatial information of radar point clouds, and realizes accurate perception of the operating environment situation information based on the cross-attention mechanism.

[0061] The communication-supported vehicle-road heterogeneous traffic subject collaborative perception method mainly includes high-precision positioning perception and beyond-line-of-sight environmental perception. High-precision positioning perception is supported by the roadside RTK (Real-time kinematic) differential base station to ensure that the autonomous driving vehicle can achieve lane-level continuous and seamless positioning. Beyond-line-of-sight environmental perception is supported by the V2X communication network to achieve interactive sharing of multi-source perception data through vehicle-to-vehicle collaboration and vehicle-road collaboration, thereby enhancing the vehicle's perception of traffic environment information.

[0062] The cloud-edge distributed perception computing architecture is used to process vehicle-mounted and roadside perception data, and reasonably distributes the data to the corresponding servers based on the size and urgency of the computing demand.

[0063] Swarm intelligent decision-making and optimization based on collaborative perception enhancement uses the vehicle's own position information and obstacle information in the driving environment obtained through vehicle-to-vehicle and vehicle-road collaboration to solve swarm intelligence problems based on the concept of multiple low-intelligence agents collaborating to achieve high-level intelligent behavior in bionics, and optimize the performance of decision-making technologies such as path search and planning for autonomous driving vehicles.

[0064] The embodiment of the present invention provides a processing flow of a method for collaborative beyond-visual-range perception and group intelligent decision-making of heterogeneous traffic entities. Figure 2 As shown, the processing steps include the following:

[0065] Step S1: The autonomous driving main vehicle adaptively selects data input according to current weather conditions and determines the type of vehicle-mounted sensor for fusion perception.

[0066] The autonomous driving main vehicle extracts cross-modal and multi-scale features of the on-board camera images and radar point cloud data, and implements deep fusion perception calculation based on the transformer cross-attention mechanism through sensors to obtain deep fusion perception data.

[0067] The deep fusion method based on the Transformer cross-attention mechanism extracts keys and values ​​from image data rich in semantic texture information, generates queries from sparse and disordered point cloud data with three-dimensional spatial information, and then uses high-resolution visual images to guide low-resolution radar point clouds to complete environmental perception tasks such as target detection. The calculation formula for cross-attention is as follows:

[0068] Q=W Q XL ,K=W K X C ,V=W V X C

[0069]

[0070]

[0071] Where Q, K, and V are query, key, and value matrices respectively, and W Q , W K , W V are their corresponding weight parameters, X L , X C are radar and camera sensor data inputs respectively, α is the normalized dot product attention calculation, CrossAttention is the cross attention calculation, and d k is the dimension of K vector;

[0072] The position encoding of the Transformer cross-attention mechanism network uses a two-dimensional rotation encoding as shown below:

[0073]

[0074] In the formula, θ i =10000 -2i / d , and d is the dimension of the feature vector.

[0075] After building the Transformer cross-attention mechanism network model, data needs to be collected for training and verification to determine the parameters of the situational deep fusion network. The Transformer cross-attention mechanism network includes a backbone network, a multi-scale feature extraction layer, a fusion network, and a detection head. The backbone network uses ResNet50, which mainly extracts image features; the multi-scale feature extraction layer mainly selects the effective features extracted by the backbone network to fully obtain data information at different scales; the fusion network realizes the deep fusion of radar data features according to the encoding and decoding structure; the detection head outputs the results of perception tasks such as prediction boxes and category confidence.

[0076] Step S2: Based on the constraints of the communication range of the autonomous driving main vehicle, determine the coordinated heterogeneous traffic entities such as vehicles and roads. The above-mentioned heterogeneous traffic entities include adjacent vehicles and intelligent roadside equipment. The roadside equipment mainly refers to digital transportation infrastructure with autonomous perception, computing and networked communication capabilities, such as smart traffic lights, smart lamp poles, etc.

[0077] Determine whether it is the first collaboration between the autonomous driving main vehicle and the heterogeneous traffic subject. If so, the heterogeneous traffic subject transmits the original sensor data collected by its sensors to the cloud computing server; otherwise, the heterogeneous traffic subject transmits the deep fusion perception data to the main vehicle.

[0078] Figure 3 A C-V2X-supported vehicle-road heterogeneous traffic subject collaborative perception diagram is provided for an embodiment of the present invention. Positioning perception in collaborative perception is mainly based on vehicle-mounted satellite navigation, inertial navigation, and camera sensors. The intelligent roadside provides differential correction numbers for the autonomous driving main vehicle to achieve lane-level high-precision positioning. For traffic environment perception, when the heterogeneous traffic subject performs collaborative perception with the main vehicle for the first time, the main vehicle and the heterogeneous traffic subject transmit their own original sensor information to the cloud computing server based on C-V2X Uu. The cloud computing server uses the ICP (Iterative Closest Point) algorithm and other algorithms to realize the collaboration of data from the perspective of the main vehicle, and then realizes the fusion of radar data and the solution of perception results based on the cross-attention mechanism. The cloud computing server transmits the calculated vehicle-road heterogeneous traffic subject collaborative perception results to the main vehicle. Go to step S4.

[0079] When the heterogeneous traffic subject is not the first time to perform cooperative perception with the main vehicle, the subsequent cooperative perception only transmits the perception results and the corresponding coordinate transformation matrix. The adjacent vehicle or intelligent roadside first fuses the data of its own radar and camera according to the cross attention mechanism, and transmits the perceived obstacle information to the main vehicle through C-V2XPC5. Execute step S3.

[0080] Step S3: If the number of coordinated heterogeneous traffic entities is greater than or equal to 3, the cloud computing center provides computing resource support to assist the autonomous driving vehicle in completing the beyond-visual-range collaborative perception calculation. If the number of coordinated heterogeneous traffic entities is less than 3, the autonomous driving vehicle uses the vehicle terminal and the edge computing center to complete the perception solution.

[0081] From the perspective of the autonomous driving vehicle, the multi-source data of the autonomous driving vehicle and other collaborative heterogeneous traffic entities are integrated to achieve beyond-visual-range collaborative perception among multiple traffic entities. First, the multi-source data is temporally and spatially aligned. The satellite positioning clock source is used to provide a time reference, and the time alignment is achieved based on the timestamp and interpolation extrapolation method of the multi-source data. The spatial alignment is achieved based on the relationship between the local coordinate system of the autonomous driving vehicle, the local coordinate systems of other heterogeneous traffic entities, and the global coordinate system. The specific process is as follows:

[0082] G V =[R V T V ]L V

[0083] G O =[RO T O ]L O

[0084] L VO =[R V T V ] -1 [R O T O ]L O

[0085] In the formula, G V is the perception data of the main vehicle in the global coordinate system, L V G is the perception data of the main vehicle in the local coordinate system of the main vehicle, O is the perception data of other coordinated heterogeneous traffic entities in the global coordinate system, L O is the self-perception data of other coordinated heterogeneous traffic entities in the local coordinate system, R V , R O and T V 、T O are the corresponding rotation and translation matrices, L VO It is the perception data of other coordinated heterogeneous traffic entities in the local coordinate system of the main vehicle.

[0086] Then, the results of the collaborative perception of heterogeneous traffic entities are determined by combining the intersection-and-joint ratio and confidence. When the intersection-and-joint ratio of the perception results between heterogeneous multi-traffic entities is greater than the threshold N, it is considered that the perceived target is the same, and the results with higher confidence are retained; otherwise, they are retained. Based on the above process, the results of the collaborative perception of heterogeneous traffic entities, including the position, speed, direction, and traffic signs of obstacles and other road participants in the traffic environment, are obtained.

[0087] In addition, at a signalized intersection, the main vehicle can also receive signal phase information transmitted by the intelligent traffic light, which is used for the vehicle to fully consider the situation of passing through the intersection when optimizing the decision on how to reach the destination.

[0088] Step S4: The host vehicle uses a decision-making algorithm based on swarm intelligence cooperative evolution according to the collaborative perception results of the heterogeneous traffic entities to obtain the optimal driving decision result of the host vehicle.

[0089] On the one hand, according to the beyond-visual-range collaborative perception results of heterogeneous traffic entities and the path nodes involved, the driving environment is uniformly represented, and decisions such as acceleration, deceleration and steering are made based on information such as the location of obstacles. On the other hand, the collaborative heterogeneous traffic entities are regarded as ant colonies, and a decision-making algorithm based on the cooperative evolution of swarm intelligence is proposed to support the main vehicle in obtaining the optimal driving path. Each entity simulates the pheromone release and volatility characteristics of ants during the driving process, and it iteratively updates the transfer road and path weight parameters of the heterogeneous traffic entities through the continuous changes in the speed and position of individuals in the heterogeneous traffic group, and uses swarm intelligence to obtain the optimal path information; finally, the optimal decision results such as the driving path are output.

[0090] Determine the driving purpose of the autonomous driving main vehicle, obtain the potential feasible solutions of all optimal paths for the main vehicle to reach the destination, define the necessity and optimality of each path as the path weight. The larger the path weight, the more likely the main vehicle should choose this path. Then, initialize the path weight parameters.

[0091] The transfer probability of heterogeneous traffic entities under the current traffic environment is iteratively calculated according to the path weight parameter, and the possibility of heterogeneous traffic entities from the current position to the next path node is determined as follows:

[0092]

[0093] Where t is the current time, P ij is the transition probability of a heterogeneous traffic subject from the current location i to the next path node j via the path, ω ij (t) is the path weight parameter, is the safety risk factor of path ij obtained based on collaborative sensing information, is the sum of the calculation results of all possible path nodes, α is the path weight importance factor, and β is the collaborative perception information importance factor.

[0094] Analogous to the excitation and inhibition principle of pheromone release during the foraging and returning to the nest behavior of ants, when passing the optimal path, the weight parameter of the corresponding path is increased by simulating the pheromone release process, and when passing the worst path, the weight parameter of the corresponding path is reduced by simulating the pheromone volatilization process. The iterative update principle of the path weight parameter is as follows:

[0095]

[0096] In the formula, ρ is the weight decay parameter, is the weight of the mth coordinated vehicle’s contribution to path ij, αω ij is the sum of the weight increments of path ij at time t, where:

[0097]

[0098] Where k is the kth iteration, L k,m is the current path distance of the mth vehicle, L min is the optimal path distance, L max is the worst path distance, and ε is the threshold constant.

[0099] Determine whether the maximum number of iterations has been reached. If not, continue the above iterative calculation process; otherwise, output the optimal decision result of the autonomous driving main vehicle based on swarm intelligence, that is, after fully considering the perception information of the previous traffic environment, the current acceleration, deceleration, steering and other decisions are solved based on the collaboration of heterogeneous traffic entities, and the optimal driving path to the destination is determined based on the size of the path weight parameter.

[0100] Collaborative perception of heterogeneous traffic entities on vehicles and roads includes: when major traffic accidents and severe congestion occur on the road, the relevant information will be perceived by the digital traffic infrastructure and the expected congestion time will be calculated, and then the relevant data will be transmitted to the cloud. The cloud can transmit the perception and calculation results to the connected vehicles within the coverage area, thereby improving the decision-making and planning performance of the connected vehicles.

[0101] Figure 4 A beyond-visual-range collaborative perception and group intelligent decision-making scenario diagram provided by an embodiment of the present invention.

[0102] exist Figure 4 In the lateral beyond-visual-range perception scenario shown in the figure, the main vehicle's field of vision is blocked by the adjacent large truck, and it is unable to fully perceive the actual traffic environment information of the intersection. There is a risk of collision between the undetected cars at the intersection and the main vehicle. However, through interaction with intelligent roadside equipment, a perspective effect can be achieved to realize lateral beyond-visual-range perception enhancement. When the main vehicle makes decision-making plans, it can fully consider the traffic situation at the intersection to obtain the expected speed and path information.

[0103] exist Figure 4 In the longitudinal beyond-visual-range perception scenario shown in , the main vehicle cannot perceive the traffic accident ahead on the road based on the on-board sensors due to the long distance, which will affect the main vehicle's autonomous decision-making performance such as optimal path search. In the networked information interaction environment, the main vehicle receives the perception information transmitted by the front vehicle to achieve longitudinal beyond-visual-range perception enhancement, and then reasonably plans the path to avoid road congestion caused by traffic accidents, thereby realizing interactive and collaborative group intelligent decision-making.

[0104] Among them, the effect of coordinated beyond-visual-range perception of heterogeneous traffic entities such as vehicles and roads is as follows: Figure 5 shown.

[0105] In summary, the method and system for collaborative over-the-horizon perception and group intelligent decision-making of heterogeneous traffic entities proposed in the present invention realizes the deep fusion perception of radar point cloud and visual image, and the comprehensive and accurate perception of the self-position and traffic environment situation information, which can solve the shortcomings of limited vehicle-mounted autonomous perception distance and the existence of perception blind spots, provide a high-quality data basis for vehicle autonomous decision-making control, and promote the formation of a group intelligent decision-making theoretical system through the over-the-horizon collaborative perception of heterogeneous traffic entities on the road and the optimization of path search. Based on the accurate identification of vehicle collision risks, the occurrence of traffic accidents can be effectively reduced, and the vehicle operation efficiency can be effectively improved through the optimal path search.

[0106] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0107] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0108] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0109] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for collaborative beyond-visual-range perception and group intelligent decision-making of heterogeneous traffic entities, characterized in that: include: The main vehicle extracts cross-modal and multi-scale features of the on-board camera image and radar point cloud data, implements deep fusion perception calculation based on the transformer cross-attention mechanism, and obtains deep fusion perception data; Based on the constraints of the communication range of the main vehicle, the coordinated heterogeneous traffic subjects are determined, and the heterogeneous traffic subjects adaptively transmit raw data or deeply fused perception data to the main vehicle according to the coordination rules; From the perspective of the autonomous driving vehicle, the multi-source data of the vehicle and other coordinated heterogeneous traffic entities are integrated to achieve beyond-visual-range coordinated perception among multiple traffic entities, and obtain coordinated perception results of heterogeneous traffic entities; The main vehicle adopts a decision-making algorithm based on swarm intelligence according to the collaborative perception result of the heterogeneous traffic entities to obtain the optimal driving decision result of the main vehicle; The above-mentioned method of realizing beyond-visual-range collaborative perception among multiple traffic entities by integrating multi-source data of the main vehicle and other collaborative heterogeneous traffic entities from the perspective of the autonomous driving main vehicle and obtaining collaborative perception results of heterogeneous traffic entities includes: If the number of coordinated heterogeneous traffic entities is greater than or equal to 3, the cloud computing center provides computing resource support to assist the autonomous driving vehicle in completing beyond-visual-range collaborative perception computing; If the number of coordinated heterogeneous traffic entities is less than 3, the autonomous driving vehicle uses the on-board terminal and the edge computing center to complete the perception solution; From the perspective of the autonomous driving vehicle, the multi-source data of the autonomous driving vehicle and other coordinated heterogeneous traffic entities are integrated to achieve beyond-visual-range collaborative perception among multiple traffic entities. The multi-source data is temporally and spatially aligned, and the satellite positioning clock source is used to provide a time reference. The time stamp and interpolation and extrapolation method of the multi-source data are used to achieve time alignment. The spatial alignment is achieved based on the relationship between the local coordinate system of the autonomous driving vehicle, the local coordinate systems of other heterogeneous traffic entities, and the global coordinate system. The specific process is as follows: G V =[R V T V ]L V G O =[R O T O ]L O L VO =[R V T V ] -1 [R O T O ]L O In the formula, G V is the perception data of the main vehicle in the global coordinate system, L V G is the perception data of the main vehicle in the local coordinate system of the main vehicle, O is the perception data of other coordinated heterogeneous traffic entities in the global coordinate system, L O is the self-perception data of other coordinated heterogeneous traffic entities in the local coordinate system, R V and R O is the corresponding rotation matrix, T V and T O are the corresponding translation matrices, L VO The perception data of other coordinated heterogeneous traffic entities in the local coordinate system of the main vehicle; The collaborative perception results of heterogeneous traffic entities of vehicles and roads are determined by combining the intersection-and-joint ratio and confidence. When the intersection-and-joint ratio of the perception results between heterogeneous multiple traffic entities is greater than the threshold N, it is considered that the perceived targets are the same, and the results with higher confidence are retained; otherwise, they are all retained. Based on the above processing process, the main vehicle obtains the collaborative perception results of heterogeneous traffic entities including the position, speed, direction of obstacles and other road participants in the traffic environment, as well as traffic sign information.

2. The method according to claim 1, characterized in that The main vehicle extracts cross-modal multi-scale features of the on-board camera image and radar point cloud data, implements deep fusion perception calculation based on the transformer cross attention mechanism, and obtains deep fusion perception data, including: The main vehicle extracts keys and values ​​from image data rich in semantic texture information through a deep fusion method based on a transformer cross-attention mechanism using on-board sensors, generates queries using point cloud data with sparse and disordered three-dimensional spatial information, and uses high-resolution visual images to guide low-resolution radar point clouds to complete target detection environment perception tasks, thereby achieving deep fusion perception calculations and obtaining deep fusion perception data. The transformer cross attention mechanism includes a backbone network, a multi-scale feature extraction layer, a fusion network and a detection head, wherein the backbone network uses ResNet50 to extract image features; the multi-scale feature extraction layer selects the effective features extracted by the backbone network to obtain data information of different scales; the fusion network realizes the deep fusion of the radar data features according to the encoding and decoding structure; the detection head outputs the results of the prediction box and category confidence perception task; The calculation formula based on the transformer cross attention mechanism is as follows: Q=W Q X L ,K=W K X C ,V=W V X C Where Q, K, and V are query, key, and value matrices respectively, and W Q , W K , W V are their corresponding weight parameters, X L , X C are radar and camera sensor data inputs respectively, α is the normalized dot product attention calculation, CrossAttention is the cross attention calculation, and d k is the dimension of K vector; The position encoding of the Transformer cross-attention mechanism uses a two-dimensional rotation encoding as shown below: In the formula, θ i =10000 -2i / d , and d is the dimension of the feature vector.

3. The method according to claim 2, characterized in that The method of determining the coordinated heterogeneous traffic subject based on the constraint of the communication range of the main vehicle, and the heterogeneous traffic subject adaptively transmitting the original data or the deeply fused perception data to the main vehicle according to the coordination rules, includes: Based on the constraints of the communication range of the autonomous driving main vehicle, determine the coordinated vehicle-road heterogeneous traffic entities, the heterogeneous traffic entities include adjacent vehicles and intelligent roadside equipment, and the roadside equipment includes digital traffic infrastructure with autonomous perception, computing and networked communication capabilities; Determine whether it is the first collaboration between the autonomous driving main vehicle and the heterogeneous traffic subject. If so, the heterogeneous traffic subject transmits the original sensor data collected by its sensors to the cloud computing server; otherwise, the heterogeneous traffic subject transmits the deep fusion perception data to the main vehicle.

4. The method according to claim 1, characterized in that: The host vehicle uses a decision-making algorithm based on swarm intelligence to obtain the optimal driving decision result of the host vehicle according to the collaborative perception result of the heterogeneous traffic entities, including: According to the beyond-visual-range collaborative perception results of heterogeneous traffic entities and the path nodes involved, the driving environment is uniformly characterized through a decision-making algorithm based on swarm intelligence, and the collaborative heterogeneous traffic entities are regarded as ant colonies. Each heterogeneous traffic entity simulates the pheromone release and volatilization characteristics of ants during the driving process. The transfer paths and path weight parameters of the heterogeneous traffic entities are iteratively updated through the continuous changes in the speed and position of the heterogeneous traffic entities, and the optimal path information is obtained by using swarm intelligence. Finally, the optimal driving decision result of the main vehicle is output, and the optimal driving decision result includes the acceleration, deceleration and steering decisions of the main vehicle at the current moment, as well as the optimal driving path to reach the destination.

5. The method according to claim 4, characterized in that The host vehicle uses a swarm intelligence-based decision algorithm to obtain an optimal driving decision result of the host vehicle according to the collaborative perception result of the heterogeneous traffic entities, and further includes: Determine the driving purpose of the autonomous driving vehicle, obtain the potential feasible solutions of all optimal paths for the vehicle to reach the destination, define the necessity and optimality of each path as the path weight, the larger the path weight, the more likely the vehicle should choose this path, and initialize the path weight parameters; The transfer probability of heterogeneous traffic entities under the current traffic environment is iteratively calculated according to the path weight parameter, and the possibility of heterogeneous traffic entities from the current position to the next path node is determined as follows: Where t is the current time, P ij is the transition probability of the heterogeneous traffic subject from the current position i to the next path node j, ω ij (t) is the path weight parameter, is the safety risk factor of path ij obtained based on collaborative sensing information, is the sum of the calculation results of all possible path nodes, α is the path weight importance factor, and β is the collaborative perception information importance factor; The principles for iterative updating of path weight parameters are as follows: In the formula, ρ is the weight decay parameter, is the weight of the mth coordinated vehicle’s contribution to path ij, Δω ij is the sum of the weight increments of path ij at time t; Determine whether the maximum number of iterations has been reached. If not, continue the above-mentioned iterative calculation process; otherwise, output the optimal driving decision result of the host vehicle based on swarm intelligence.

Images