Vehicle and road cloud real-time cooperative sensing method and device oriented to intelligent network connection environment

Abstract

The invention relates to a vehicle and road cloud real-time cooperative sensing method and device for an intelligent network connection environment, and the method comprises the steps: obtaining multi-frame vehicle end and road end point cloud data of vehicle end and road end information nodes, inputting a vehicle end and road end sensing model to generate vehicle end and road end sensing results, and carrying out the fusion of the vehicle end and road end sensing results through an edge cloud; vehicle cloud and road cloud transmission time delay between the vehicle-road end and the edge cloud end is obtained to calculate fusion output total time length, and real-time sensing performance indexes are calculated in combination with sensing output lag parameters; and based on the real-time perception performance index and the real-time dynamic algorithm optimization strategy, obtaining a target optimization vehicle end and road end perception model so as to generate a vehicle-road cloud real-time cooperative perception result. Therefore, the problems that in the prior art, modeling cannot be carried out on communication time delay of transmission of sensing information to cloud in an intelligent network connection automobile cloud control system from the algorithm perspective, mutual restriction of calculation errors, time delay and the like exists in a collaborative sensing model, and models of different scales need to be used for different environment complexity are solved.

Description

Technical Field

[0001] The present application relates to the field of vehicle-road-cloud collaborative perception technology, and in particular to a vehicle-road-cloud real-time collaborative perception method and device for an intelligent network environment. Background Art

[0002] Intelligent connected vehicles use on-board sensors, controllers, actuators and other devices, combined with modern communication and network technologies, to achieve intelligent information sharing between vehicles, vehicles and roads, and vehicles and the cloud. The cloud control system of intelligent connected vehicles generates real-time digital mapping and digital twins of smart cars and driving environments in the real world in the cloud through perception and positioning communication technology, and performs various functional applications in the cloud that are safe, energy-saving and efficient, and then issues instructions. This process couples the physical world and the information world, and is a closed-loop control process on the physical side and the information side. In this system, perception needs to have both low latency in communication and computing and high accuracy in mapping the physical world to the information world.

[0003] However, existing vehicle-road cooperative perception algorithms mainly focus on the offline performance of the algorithms, ignoring the real-time issues of the algorithms in practical applications. This leads to the following problems in real-time applications such as digital twins in the cloud control system of intelligent connected vehicles:

[0004] 1. The communication delay of transmitting perception information to the cloud in the cloud control system of intelligent connected vehicles has not been modeled from an algorithmic perspective.

[0005] At present, the perception scheme design of the collaborative perception algorithm only provides the information interaction relationship and algorithm processing process for different perception nodes, and rarely considers the collaborative perception algorithm with cloud participation. It also does not consider the impact of communication delay on the processing delay of specific applications in the vehicle-road-cloud integrated system, and does not provide a clear solution to the instability of cloud communication.

[0006] 2. The collaborative perception model has mutual constraints between computational error, delay and error, and the model scale needs to be dynamically optimized for different environmental complexities.

[0007] The existing collaborative perception method based on target-level algorithm has the advantage of good real-time performance and is more suitable for the information-physical system of intelligent vehicles. However, its perception accuracy is average, and there is no way to adjust the algorithm according to the real-time dynamic environment after deployment, resulting in poor performance.

[0008] In summary, the existing technology is unable to model the communication delay of transmitting perception information to the cloud in the cloud control system of intelligent connected vehicles from an algorithmic perspective, and the collaborative perception model has the mutual constraints of computational errors and delays and errors. Models of different scales need to be used for different environmental complexities, which needs to be solved urgently. Summary of the invention

[0009] The present application provides a vehicle-road-cloud real-time collaborative perception method and device for an intelligent connected environment, in order to solve the problems that the prior art cannot model the communication delay of transmitting perception information from the intelligent connected vehicle cloud control system to the cloud from an algorithmic perspective, and that the collaborative perception model has mutual constraints such as computational errors and delays, and that models of different scales need to be used for different environmental complexities.

[0010] The first aspect of the present application provides a vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment, comprising the following steps: respectively obtaining multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in a target traffic scene, and inputting the multi-frame vehicle-end point cloud data and the multi-frame road-end point cloud data into a preset vehicle-end perception model and a road-end perception model, respectively, to generate a vehicle-end perception result and a road-end perception result; fusing the vehicle-end perception result and the road-end perception result through a target edge cloud to obtain a collaborative perception result and a fusion duration, and respectively obtaining the at least one target vehicle-end information node and the at least one road-end information node. The vehicle-cloud transmission delay and road-cloud transmission delay between the point and the target edge cloud are calculated to calculate the total fusion output time based on the vehicle-cloud transmission delay, the road-cloud transmission delay and the fusion time, and the real-time perception performance index corresponding to the collaborative perception result is calculated according to the total fusion output time and the preset perception output lag parameter; based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy, the vehicle-side perception model and the road-side perception model are subjected to preset dynamic optimization operations to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model, and the vehicle-road-cloud real-time collaborative perception result is generated according to the target optimized vehicle-side perception model and the target optimized road-side perception model.

[0011] Optionally, in one embodiment of the present application, the multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in the target traffic scene are respectively obtained, and the multi-frame vehicle-end point cloud data and the multi-frame road-end point cloud data are respectively input into a preset vehicle-end perception model and a road-end perception model to generate a vehicle-end perception result and a road-end perception result, including: respectively obtaining multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data through the at least one target vehicle-end information node and the at least one road-end information node, and using the vehicle-end perception model to detect and track the multi-frame vehicle-end point cloud data to generate the vehicle-end perception result; using the road-end perception model to perform road-end target detection and feature extraction operations on the multi-frame road-end point cloud data to generate the road-end perception result.

[0012] Optionally, in one embodiment of the present application, the fusion of the vehicle-side perception results and the road-side perception results through the target edge cloud to obtain a collaborative perception result and a fusion duration includes: performing coordinate transformation processing on the vehicle-side perception results and the road-side perception results through the target edge cloud, and performing a target-level information fusion detection operation on the vehicle-side perception results and the road-side perception results after the coordinate transformation processing to generate the collaborative perception result and the fusion duration.

[0013] Optionally, in one embodiment of the present application, the vehicle-to-cloud transmission delay and the road-to-cloud transmission delay between the at least one target vehicle-side information node and the at least one road-side information node and the target edge cloud are respectively obtained to calculate the total fusion output time based on the vehicle-to-cloud transmission delay, the road-to-cloud transmission delay and the fusion time, and the real-time perception performance index corresponding to the collaborative perception result is calculated according to the total fusion output time and the preset perception output lag parameter, including: determining the road-side model perception time corresponding to the vehicle-side perception model and the vehicle-side model perception time corresponding to the road-side perception model; according to the road-side model perception time, the vehicle-side model perception time, the vehicle-to-cloud transmission delay, the road-to-cloud transmission delay and the fusion time The total duration of the fusion output is calculated according to the time length; the total duration of the fusion output is compared with the perception output lag parameter; when the total duration of the fusion output is less than the perception output lag parameter, the first perception result performance corresponding to each frame of the vehicle endpoint cloud data in the multi-frame vehicle endpoint cloud data is calculated; when the total duration of the fusion output is greater than or equal to the perception output lag parameter, the second perception result performance corresponding to each frame of the vehicle endpoint cloud data in the multi-frame vehicle endpoint cloud data is calculated; the total number of frames of the multi-frame vehicle endpoint cloud data is obtained, and the sum of the real-time perception performance corresponding to the multi-frame vehicle endpoint cloud data is calculated according to the first perception result performance and the second perception result performance, and the real-time perception performance index is calculated based on the sum of the real-time perception performance and the total number of frames.

[0014] Optionally, in one embodiment of the present application, based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy, the vehicle-side perception model and the road-side perception model are subjected to a preset dynamic optimization operation to obtain a target optimized vehicle-side perception model and a target optimized road-side perception model, including: performing feature engineering environmental feature modeling on the target traffic scene corresponding to each frame of collaborative perception results in the multi-frame collaborative perception results to obtain an environmental feature vector corresponding to the target traffic scene; constructing multiple groups of model parameter sets of different scales for the vehicle-side perception model and the road-side perception model, respectively, and determining the AP index corresponding to each group of model parameter sets of different scales in the multiple groups of model parameter sets of different scales; inputting the vehicle-cloud transmission delay, the road-cloud transmission delay, the environmental feature vector and the AP index pair corresponding to the first frame of collaborative perception results in the multi-frame collaborative perception results into a pre-trained regression prediction model to generate target model parameters in the multiple groups of model parameter sets of different scales that meet the preset real-time perception performance index requirements; after a preset time interval , update the vehicle-cloud transmission delay, the road-cloud transmission delay, the environmental feature vector and the AP index according to the vehicle-side perception model and the road-side perception model corresponding to the target model parameters to obtain a new vehicle-cloud transmission delay, a new road-cloud transmission delay, a new environmental feature vector and a new AP index corresponding to the second frame collaborative perception result, and input the new vehicle-cloud transmission delay, the new road-cloud transmission delay, the new environmental feature vector, the new AP index and the target model parameters into the regression prediction model to generate a new target model parameter that meets the preset real-time perception performance index requirements in the multiple groups of model parameters of different scales; iteratively execute the update operation of the vehicle-cloud transmission delay, the road-cloud transmission delay, the environmental feature vector, the AP index and the target model parameters until the target model parameters corresponding to each frame of collaborative perception results are generated, so as to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model corresponding to each frame of collaborative perception results based on the target model parameters corresponding to each frame of collaborative perception results.

[0015] The second aspect of the present application provides a vehicle-road-cloud real-time collaborative perception device for an intelligent network environment, including: a perception module, used to respectively obtain multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in a target traffic scene, and input the multi-frame vehicle-end point cloud data and the multi-frame road-end point cloud data into a preset vehicle-end perception model and a road-end perception model, respectively, to generate a vehicle-end perception result and a road-end perception result; an indicator construction module, used to fuse the vehicle-end perception result and the road-end perception result through a target edge cloud to obtain a collaborative perception result and a fusion duration, and respectively obtain the at least one target vehicle-end information node and the at least one road-end information node. The vehicle-cloud transmission delay and road-cloud transmission delay between the node and the target edge cloud are used to calculate the total fusion output time based on the vehicle-cloud transmission delay, the road-cloud transmission delay and the fusion time, and the real-time perception performance index corresponding to the collaborative perception result is calculated according to the total fusion output time and the preset perception output lag parameter; a dynamic optimization module is used to perform preset dynamic optimization operations on the vehicle-side perception model and the road-side perception model based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model, and generate the vehicle-road-cloud real-time collaborative perception result according to the target optimized vehicle-side perception model and the target optimized road-side perception model.

[0016] Optionally, in one embodiment of the present application, the perception module includes: an acquisition unit, used to respectively acquire multiple frames of vehicle-end point cloud data and multiple frames of road-end point cloud data through the at least one target vehicle-end information node and the at least one road-end information node, and use the vehicle-end perception model to detect and track the multiple frames of vehicle-end point cloud data to generate the vehicle-end perception result; a feature extraction unit, used to use the road-end perception model to perform road-end target detection and feature extraction operations on the multiple frames of road-end point cloud data to generate the road-end perception result.

[0017] Optionally, in one embodiment of the present application, the indicator construction module includes: a fusion unit, used to perform coordinate conversion processing on the vehicle-side perception results and the road-side perception results through the target edge cloud, and perform target-level information fusion detection operations on the vehicle-side perception results and the road-side perception results after the coordinate conversion processing to generate the collaborative perception results and the fusion duration.

[0018] Optionally, in one embodiment of the present application, the indicator construction module also includes: a first determination unit, used to determine the roadside model perception time corresponding to the vehicle-side perception model and the vehicle-side model perception time corresponding to the road-side perception model; a first calculation unit, used to calculate the total fusion output time according to the roadside model perception time, the vehicle-side model perception time, the vehicle-cloud transmission delay, the road-cloud transmission delay and the fusion time; a comparison unit, used to compare the total fusion output time and the perception output lag parameter; a second calculation unit, used to calculate the fusion output total time when the fusion output total time is less than the perception output lag parameter The first perception result performance corresponding to each frame of vehicle endpoint cloud data in the multiple frames of vehicle endpoint cloud data is calculated; the third calculation unit is used to calculate the second perception result performance corresponding to each frame of vehicle endpoint cloud data in the multiple frames of vehicle endpoint cloud data when the total duration of the fusion output is greater than or equal to the perception output lag parameter; the fourth calculation unit is used to obtain the total number of frames of the multiple frames of vehicle endpoint cloud data, and calculate the sum of real-time perception performance corresponding to the multiple frames of vehicle endpoint cloud data according to the first perception result performance and the second perception result performance, and calculate the real-time perception performance index based on the sum of real-time perception performance and the total number of frames.

[0019] Optionally, in one embodiment of the present application, the dynamic optimization module includes: a modeling unit, which is used to perform feature engineering environmental feature modeling on the target traffic scene corresponding to each frame of collaborative perception results in the multi-frame collaborative perception results, so as to obtain the environmental feature vector corresponding to the target traffic scene; a second determination unit, which is used to construct multiple groups of model parameter sets of different scales for the vehicle-side perception model and the road-side perception model, respectively, and determine the AP index corresponding to each group of model parameter sets of different scales in the multiple groups of model parameter sets of different scales; a first generation unit, which is used to input the vehicle-cloud transmission delay, the road-cloud transmission delay, the environmental feature vector and the AP index pair corresponding to the first frame of collaborative perception results in the multi-frame collaborative perception results into a pre-trained regression prediction model, so as to generate the target model parameters in the multiple groups of model parameter sets of different scales that meet the preset real-time perception performance index requirements; a second generation unit, which is used to generate the target model parameters in the multiple groups of model parameter sets of different scales after a preset time interval according to the vehicle-side perception model and the road-side perception model corresponding to the target model parameters. The known model updates the vehicle-cloud transmission delay, the road-cloud transmission delay, the environmental feature vector and the AP index to obtain a new vehicle-cloud transmission delay, a new road-cloud transmission delay, a new environmental feature vector and a new AP index corresponding to the collaborative perception result of the second frame, and inputs the new vehicle-cloud transmission delay, the new road-cloud transmission delay, the new environmental feature vector, the new AP index and the target model parameters into the regression prediction model to generate a new target model parameter that meets the preset real-time perception performance index requirements among the multiple groups of model parameters of different scales; an iteration unit is used to iteratively execute the update operation of the vehicle-cloud transmission delay, the road-cloud transmission delay, the environmental feature vector, the AP index and the target model parameters until the target model parameters corresponding to each frame of collaborative perception results are generated, so as to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model corresponding to each frame of collaborative perception results based on the target model parameters corresponding to each frame of collaborative perception results.

[0020] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment as described in the above embodiments.

[0021] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment.

[0022] The fifth aspect embodiment of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment.

[0023] Therefore, the embodiments of the present application have the following beneficial effects:

[0024] The embodiments of the present application can respectively obtain multi-frame vehicle-end cloud data and multi-frame road-end cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in the target traffic scene, and input the multi-frame vehicle-end cloud data and the multi-frame road-end cloud data into a preset vehicle-end perception model and a road-end perception model, respectively, to generate a vehicle-end perception result and a road-end perception result; fuse the vehicle-end perception result and the road-end perception result through the target edge cloud to obtain a collaborative perception result and a fusion time, and respectively obtain the information between at least one target vehicle-end information node and at least one road-end information node and the target edge cloud. The vehicle-cloud transmission delay and the road-cloud transmission delay are used to calculate the total fusion output time based on the vehicle-cloud transmission delay, the road-cloud transmission delay and the fusion time, and the real-time perception performance index corresponding to the collaborative perception result is calculated based on the total fusion output time and the preset perception output lag parameter; based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy, the vehicle-side perception model and the road-side perception model are preset for dynamic optimization operations to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model, and the vehicle-road-cloud real-time collaborative perception result is generated based on the target optimized vehicle-side perception model and the target optimized road-side perception model. The present application can improve the real-time and accurate performance of the target-level collaborative perception algorithm through dynamic adjustment strategies. Thus, it solves the problems that the prior art cannot model the communication delay of the transmission of perception information to the cloud in the cloud control system of intelligent connected vehicles from an algorithmic perspective, and the collaborative perception model has mutual constraints such as calculation errors and delays, and different scales of models are required for different environmental complexities.

[0025] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A flow chart of a vehicle-road-cloud real-time collaborative perception method for an intelligent network environment provided according to an embodiment of the present application;

[0028] Figure 2 A schematic diagram of a real-time algorithm framework for vehicle-road-cloud collaborative perception provided for one embodiment of the present application;

[0029] Figure 3 A schematic diagram of a target-level vehicle-road-cloud collaborative target detection algorithm framework provided for an embodiment of the present application;

[0030] Figure 4 A schematic diagram of the execution logic of a point cloud-based target-level vehicle-road-cloud collaborative target detection algorithm provided for one embodiment of the present application;

[0031] Figure 5 A schematic diagram of a calculation process of a vehicle-road-cloud collaborative perception algorithm provided for an embodiment of the present application;

[0032] Figure 6 A schematic diagram of collaborative sensing calculation results under extreme time delay conditions provided for an embodiment of the present application;

[0033] Figure 7 A schematic diagram of a logical architecture of a dynamic optimization strategy provided for an embodiment of the present application;

[0034] Figure 8 An embodiment of the present application provides a conservative With f globalbest Schematic diagram of algorithm operation comparison;

[0035] Fig. 9 This is an example diagram of a vehicle-road-cloud real-time collaborative perception device for an intelligent network environment according to an embodiment of the present application;

[0036] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0037] Among them, 10-vehicle-road-cloud real-time collaborative perception device for intelligent connected environment; 100-perception module, 200-indicator construction module, 300-dynamic optimization module; 1001-memory, 1002-processor, 1003-communication interface. DETAILED DESCRIPTION

[0038] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0039] The following describes the vehicle-road-cloud real-time collaborative perception method and device for an intelligent connected environment in an embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides a vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment. In this method, multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in a target traffic scene are obtained respectively, and the multi-frame vehicle-end point cloud data and the multi-frame road-end point cloud data are input into a preset vehicle-end perception model and a road-end perception model respectively to generate a vehicle-end perception result and a road-end perception result; the vehicle-end perception result and the road-end perception result are fused on the target edge cloud to obtain a collaborative perception result and a fusion duration, and at least one target vehicle-end information node is obtained respectively. The vehicle-cloud transmission delay and road-cloud transmission delay between at least one road-side information node and the target edge cloud are used to calculate the total fusion output time based on the vehicle-cloud transmission delay, road-cloud transmission delay and fusion time, and the real-time perception performance index corresponding to the collaborative perception result is calculated according to the total fusion output time and the preset perception output lag parameter; based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy, the vehicle-side perception model and the road-side perception model are preset dynamically optimized to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model, and the vehicle-road-cloud real-time collaborative perception result is generated according to the target optimized vehicle-side perception model and the target optimized road-side perception model. The present application can improve the real-time and accurate performance of the target-level collaborative perception algorithm through dynamic adjustment strategies. Thus, the problems that the prior art cannot model the communication delay of the transmission of perception information to the cloud in the cloud control system of intelligent connected vehicles from an algorithmic perspective, and the collaborative perception model has mutual constraints such as calculation errors and delays, and different scales of models are required for different environmental complexities are solved.

[0040] Specifically, Figure 1 A flow chart of a vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment provided in an embodiment of the present application.

[0041] like Figure 1 As shown, the vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment includes the following steps:

[0042] In step S101, multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in the target traffic scene are respectively obtained, and the multi-frame vehicle-end point cloud data and the multi-frame road-end point cloud data are respectively input into the preset vehicle-end perception model and road-end perception model to generate vehicle-end perception results and road-end perception results.

[0043] It can be understood that in the embodiments of the present application, vehicle-road-cloud collaborative perception is a subtask of the overall perception task of the intelligent connected vehicle cloud control system in an intelligent connected environment. Its main function is to use information from different perspectives on both sides of the vehicle and road to perform collaborative perception of traffic scenes, thereby helping to achieve the task of generating digital twins and real-time mapping in the edge cloud.

[0044] It should be noted that the real-time algorithm framework of vehicle-road-cloud collaborative perception in the embodiment of the present application mainly involves the vehicle side, the road side and the edge cloud side, such as Figure 2 As shown, the embodiment of the present application has carried out a specific definition of modular information transmission and algorithm execution process according to the respective characteristics of the three ends of the vehicle, road and cloud. Specifically, due to the wireless transmission characteristics between the vehicle and the edge cloud, the original level information with high transmission load requirements cannot be transmitted between the vehicle and the edge cloud. In this embodiment of the present application, the information transmission between the vehicle and the edge cloud can be target-level information, as well as feature-level information with low load requirements after encoding and information screening; since the information communication between the road and the edge cloud is transmitted through a wired connection with a large load bearing capacity, the embodiment of the present application can be defined as a transmission that can carry original-level data.

[0045] Therefore, the embodiments of the present application can respectively obtain multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to the vehicle-end information nodes and road-end information nodes in the traffic scene, and input them into the vehicle-end perception model and the road-end perception model respectively, thereby generating vehicle-end perception results and road-end perception results.

[0046] Optionally, in one embodiment of the present application, multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in the target traffic scene are respectively obtained, and the multi-frame vehicle-end point cloud data and the multi-frame road-end point cloud data are respectively input into a preset vehicle-end perception model and a road-end perception model to generate a vehicle-end perception result and a road-end perception result, including: obtaining multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data respectively through at least one target vehicle-end information node and at least one road-end information node, and using the vehicle-end perception model to detect and track the multi-frame vehicle-end point cloud data to generate a vehicle-end perception result; using the road-end perception model to perform road-end target detection and feature extraction operations on the multi-frame road-end point cloud data to generate a road-end perception result.

[0047] In the actual execution process, the operation process of the vehicle-road-cloud collaborative perception real-time algorithm in the embodiment of the present application on the vehicle side is:

[0048] 1. K-frame vehicle endpoint cloud data acquisition;

[0049] 2. Fusion target detection and tracking: Use target detection and tracking algorithms (i.e., vehicle-side perception model) to detect and track K frames of data, respectively, and then combine the target-level data of the previous frame transmitted by the edge cloud to perform trajectory matching and update to generate vehicle-side perception results;

[0050] 3. Determine the region of interest: Determine the region of interest based on trajectory rules and the field of view information after blind spot filling on the vehicle side;

[0051] 4. Region of interest coding: Use feature extractors and feature selectors to encode regions and upload them to the edge cloud;

[0052] 5. Obtain K+1 frames of vehicle endpoint cloud data.

[0053] The execution process of the vehicle-road-cloud collaborative perception real-time algorithm in the edge cloud is as follows:

[0054] 1. Fusion of target detection and feature extraction: Extracting the perception information features of the road end;

[0055] 2. Vehicle-road target level information fusion detection and tracking: Fusion of vehicle-road information data to achieve initial ID consistent continuous tracking;

[0056] 3. Information decoding: Decode the feature-level information uploaded by the vehicle to obtain feature information;

[0057] 4. Regional feature fusion: Fusion of regional feature-level data of vehicle-road information;

[0058] 5. Target detection and reasoning tracking: Fusion detection and tracking of the region of interest, sending fusion tracking results and assisting in updating the edge cloud digital twin model.

[0059] The roadside node can obtain multi-frame road end point cloud data, and perform road-side target detection and feature extraction operations on the multi-frame road end point cloud data through the road-side perception model to generate road-side perception results.

[0060] It should be noted that the embodiment of the present application can design an algorithm framework for a vehicle-road-cloud collaborative target detection algorithm using a target-level perception algorithm based on this vehicle-road-cloud collaborative perception algorithm framework. Figure 2 The specific algorithm framework shown is Figure 3 The simplified vehicle-road-cloud collaborative perception algorithm framework shown mainly focuses on the target-level perception algorithm on both sides of the vehicle and the target-level transmission of information.

[0061] Therefore, the embodiment of the present application designs a real-time algorithm framework for vehicle-road-cloud collaborative perception, combines the characteristics of vehicle-road-cloud communication and perception, and considers communication delay and real-time perception requirements, thereby designing a target-level perception architecture for the intelligent connected vehicle cloud control system with an adaptive delay compensation mechanism, effectively realizing real-time, efficient and stable acquisition of traffic perception domain perception results.

[0062] In step S102, the vehicle-side perception results and the road-side perception results are fused through the target edge cloud to obtain the collaborative perception results and fusion time, and the vehicle-cloud transmission delay and road-cloud transmission delay between at least one target vehicle-side information node and at least one road-side information node and the target edge cloud are respectively obtained, so as to calculate the total fusion output time based on the vehicle-cloud transmission delay, the road-cloud transmission delay and the fusion time, and calculate the real-time perception performance index corresponding to the collaborative perception result according to the total fusion output time and the preset perception output lag parameter.

[0063] Furthermore, the embodiments of the present application also need to fuse the vehicle-side perception results and the road-side perception results through the edge cloud to obtain the collaborative perception results and obtain the fusion time; after that, the embodiments of the present application can respectively obtain the vehicle-to-cloud transmission delay and the road-to-cloud transmission delay between the vehicle-side information node, the road-side information node and the target edge cloud, thereby calculating the total fusion output time through the vehicle-to-cloud transmission delay, the road-to-cloud transmission delay and the fusion time, and calculating the real-time perception performance index PAP (Practical Average Precision) based on the total fusion output time and the perception output lag parameter. Define the evaluation index of the real-time algorithm of vehicle-road-cloud collaborative perception.

[0064] Therefore, the embodiment of the present application determines the real-time perception performance indicators of vehicle-road-cloud collaborative perception, thereby modeling the impact of vehicle-road-cloud communication delay and computing delay on the real-time performance of the collaborative perception algorithm.

[0065] Optionally, in one embodiment of the present application, the vehicle-side perception results and the road-side perception results are fused through the target edge cloud to obtain collaborative perception results and fusion duration, including: performing coordinate conversion processing on the vehicle-side perception results and the road-side perception results through the target edge cloud, and performing target-level information fusion detection operations on the vehicle-side perception results and the road-side perception results after the coordinate conversion processing to generate collaborative perception results and fusion duration.

[0066] It should be noted that in the fusion process of edge cloud, the embodiments of the present application can use point cloud data as input at both ends, such as Figure 4 As shown, the point cloud data at both ends of the vehicle and the road are input into their respective target detection algorithms (i.e., the vehicle-side perception model and the road-side perception model) to obtain corresponding vehicle-road target-level results (i.e., the vehicle-side perception results and the road-side perception results); afterwards, the embodiment of the present application can upload the vehicle-road target-level results to the cloud, and perform coordinate transformation and alignment on the cloud, and after target-level fusion, output the collaborative perception results, and at the same time calculate the fusion time of the fusion process on the edge cloud.

[0067] Therefore, the embodiments of the present application obtain the collaborative perception results and fusion time of the edge cloud, thereby providing reliable data guidance and basis for the realization of real-time collaborative perception of the vehicle, road and cloud.

[0068] Optionally, in one embodiment of the present application, the vehicle-to-cloud transmission delay and the road-to-cloud transmission delay between at least one target vehicle-side information node and at least one road-side information node and the target edge cloud are respectively obtained, so as to calculate the total fusion output time based on the vehicle-to-cloud transmission delay, the road-to-cloud transmission delay and the fusion time, and calculate the real-time perception performance index corresponding to the collaborative perception result according to the total fusion output time and the preset perception output lag parameter, including: determining the road-side model perception time corresponding to the vehicle-side perception model and the vehicle-side model perception time corresponding to the road-side perception model; according to the road-side model perception time, the vehicle-side model perception time, the vehicle-to-cloud transmission delay, the road-to-cloud transmission delay and the fusion time The total duration of the fusion output is calculated according to the time length; the total duration of the fusion output is compared with the perception output lag parameter; when the total duration of the fusion output is less than the perception output lag parameter, the first perception result performance corresponding to each frame of the multi-frame vehicle endpoint cloud data is calculated; when the total duration of the fusion output is greater than or equal to the perception output lag parameter, the second perception result performance corresponding to each frame of the multi-frame vehicle endpoint cloud data is calculated; the total number of frames of the multi-frame vehicle endpoint cloud data is obtained, and the sum of the real-time perception performance corresponding to the multi-frame vehicle endpoint cloud data is calculated according to the first perception result performance and the second perception result performance, and the real-time perception performance index is calculated based on the sum of the real-time perception performance and the total number of frames.

[0069] For the establishment of real-time perception performance indicators in vehicle-road-cloud collaborative perception, the embodiment of the present application first needs to determine the source of perception errors in the intelligent connected environment. The high real-time nature of the input can usually be described by the number of perception data processed per second, and is represented by the frame rate FPS (Frame Per Second), with the unit being Hz.

[0070] It should be noted that the target-level collaborative perception algorithm calculation process in the embodiment of the present application is as follows: Figure 5 As shown, (t 0+α -t 0 ) represents the two ends of the road 0 The input data is respectively passed through the target-level perception algorithm loaded on each computing platform to output the target-level data; (t 0+α+β -t 0+α ) is the matching delay, which means the target-level data transmission time in the process of target-level perception target matching domain association on the edge cloud after the target-level perception results output by the target-level perception algorithm at both ends of the vehicle-road. The above two calculation times together constitute the target-level vehicle-road-cloud collaborative perception algorithm delay (t 0+α+β -t 0 ).

[0071] In the embodiments of the present application, Figure 5 The perception results on the right are shown in the figure, with t 0 The red truck input in the vehicle-road data pair is the collaborative perception example target, and the blue target detection box is the output of this collaborative perception algorithm for t 0 The perception output result of the input data at each moment is The classification results are t 0 The perception output result of the input data at each moment is shown in the following formula:

[0072]

[0073] in, Respectively represent the geometric center point of the perception results of this truck in the x, y, and z directions; Represents the perceived size of the truck in the x, y, and z directions respectively.

[0074] Similarly, in the embodiment of the present application, the red target detection frame can be set to be when the collaborative perception algorithm is at t 0+α+β Time output The corresponding current truck type information is And the real target information As shown below:

[0075]

[0076] Understandably, Figure 5 The gap between the actual position of the target indicated by the purple arrow and the perception result output by the perception algorithm, that is, the perception error caused by the delay, can be expressed as Delays may also cause and Any discrepancy between the two is a false positive.

[0077] At the same time, if the delay is too large, it may cause Figure 6 The results shown are Figure 6 There is no overlap between the output of the collaborative perception algorithm shown in the blue box and the red box representing the actual position of the current target, which may lead to the situation where the perception results cannot be used and cannot be matched.

[0078] The following embodiments of the present application will provide a formal problem description and definition of the real-time collaborative perception problem in the research scenario described above.

[0079] It should be noted that the observation data stream in the embodiment of the present application can be modeled as a set of sensor observations, that is, a set of a series of world states and timestamps. For the 3D object detection problem, it can be expressed as:

[0080]

[0081] Among them, X j is the original data of the point cloud; Y j For X j The calibration truth value of the data; t j For data X j The timestamp of the

[0082] For an original data X with N targets, its target true value Y can be expressed as:

[0083]

[0084] in, represents the geometric center of the target in the original point cloud data, and Respectively represent the length, width and height of the target in the original point cloud data, CLS i Indicates the category of this target.

[0085] Therefore, for vehicle-road-cloud collaborative perception, the point cloud data on both sides of the vehicle and the road can be expressed as as well as They represent the data flows at the vehicle end and the road end respectively. At the same time, since there is a transmission delay process from the vehicle to the cloud and a transmission delay process from the vehicle to the road in this problem, the two transmission delay processes can be defined as as well as Among them, the former is the transmission delay from the vehicle to the cloud, and the latter is the transmission delay from the road to the cloud.

[0086] In a typical vehicle-road-cloud collaborative perception scenario, for each frame of sensor perception data from the vehicle-road perspective, the comprehensive time from acquiring environmental perception data to outputting the fused perception results on the edge cloud can be expressed as T c It is represented as follows:

[0087] T c =max{(T fv +T v ), (T fr +T r )}+T ff (5)

[0088] Among them, T fv represents the time consumption of the vehicle-side target detection algorithm; T fr represents the time consumption of the road-side target detection algorithm; T v represents the data transmission time from vehicle to edge cloud; T r represents the transmission time from the road end node to the cloud; T ffRepresents the time consumption of the fusion algorithm. The time consumption component of this process means that in the research scenario, the process of the vehicle performing perception calculations and transmitting the perception results to the cloud is in parallel with the process of the road performing perception calculations and transmitting the perception results to the cloud. The edge cloud can collect data from vehicles and roadsides and perform fusion perception.

[0089] The specific process of real-time collaborative perception can be defined as: at the real value demand output time t′ N The output of the vehicle-road-cloud collaborative perception algorithm based on transmission delay is as follows:

[0090]

[0091] The algorithm perception result of the Nth frame is defined as:

[0092]

[0093] The timestamp of the perception output is t output , which means that the algorithm can use time t N And all previous data frames are calculated, where t N =t′ N -ε, ε is the perception output hysteresis parameter artificially set according to algorithm delay, transmission delay and deployment requirements.

[0094] It should be noted that t output ≤t′ N , and in most cases t output ≠t′ N , because the result of the collaborative sensing algorithm It can be N Any output before this time is acceptable; meanwhile, the result Timestamp t output Less than t′ N The closest timestamp.

[0095] Based on the above-described vehicle-road-cloud collaborative real-time perception problem, the performance evaluation of the vehicle-road-cloud collaborative real-time perception problem is defined as the use time t N And all previous data frames are calculated, and the required perception truth value output time t′ N The perception results output before Therefore, for the Nth output, the embodiment of the present application can compare and Y N The performance of collaborative sensing algorithms is evaluated.

[0096] The embodiment of the present application is based on the above process for the real-time collaborative perception problem, so that a new evaluation index PAP can be defined, and the pseudo code of the specific evaluation is shown in the following pseudo code:

[0097]

[0098]

[0099] It should be understood by those skilled in the art that the traditional evaluation index AP (Average Precision) cannot be directly used to evaluate the performance of this real-time problem, because the true value and predicted value of the data frame involved in the calculation in the AP value evaluation method are the same frame in all cases. Specifically, for t i The original data X at the moment i Prediction data And the true value data Y i For AP, the data pair involved in AP calculation is

[0100] For the real-time collaborative perception problem, due to the limitation of perception output time ε, i The original data at the moment; for the original data X i Prediction data And the true value data Y i For example, if Y i Output time point t i+α Greater than t i +ε, then this frame data cannot be i The perception output node corresponding to time t i +ε is used. However, due to the demand for continuous perception output by downstream applications, such as digital twins, the current time point t i +εThe latest sensory data available is the one using t i-1 Original dataX i-1 Output prediction data Therefore, for the collaborative perception problem, the data pair that needs to be calculated for real-time accuracy performance in the embodiment of the present application is

[0101] Therefore, the embodiment of the present application expands the offline performance indicator AP in the real-time perception problem into PAP through the above process, thereby describing the accurate performance of the perception results that can actually be output at the perception demand output node.

[0102] In step S103, based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy, the vehicle-side perception model and the road-side perception model are subjected to preset dynamic optimization operations to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model, and the vehicle-road-cloud real-time collaborative perception results are generated according to the target optimized vehicle-side perception model and the target optimized road-side perception model.

[0103] Furthermore, the embodiments of the present application can determine a model dynamic optimization method for a real-time collaborative target detection algorithm suitable for an intelligent connected vehicle cloud control system based on the collaborative perception architecture designed above and the defined perception real-time algorithm evaluation indicators, so as to achieve real-time and accurate perception of traffic participating vehicles in the traffic domain.

[0104] Optionally, in one embodiment of the present application, based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy, the vehicle-side perception model and the road-side perception model are subjected to a preset dynamic optimization operation to obtain a target optimized vehicle-side perception model and a target optimized road-side perception model, including: performing feature engineering environmental feature modeling on the target traffic scene corresponding to each frame of the collaborative perception results in the multi-frame collaborative perception results to obtain an environmental feature vector corresponding to the target traffic scene; constructing multiple groups of model parameter sets of different scales for the vehicle-side perception model and the road-side perception model, respectively, and determining the AP index corresponding to each group of model parameter sets of different scales in the multiple groups of model parameter sets of different scales; inputting the vehicle-cloud transmission delay, road-cloud transmission delay, environmental feature vector and AP index pair corresponding to the first frame of the collaborative perception results in the multi-frame collaborative perception results into a pre-trained regression prediction model to generate target model parameters in multiple groups of model parameter sets of different scales that meet the preset real-time perception performance index requirements; After a set time interval, the vehicle-cloud transmission delay, road-cloud transmission delay, environmental feature vector and AP index are updated according to the vehicle-side perception model and road-side perception model corresponding to the target model parameters to obtain the new vehicle-cloud transmission delay, new road-cloud transmission delay, new environmental feature vector and new AP index corresponding to the second frame of collaborative perception result, and the new vehicle-cloud transmission delay, new road-cloud transmission delay, new environmental feature vector, new AP index and target model parameters are input into the regression prediction model to generate multiple groups of model parameters of different scales that meet the requirements of preset real-time perception performance indicators. The update operation of vehicle-cloud transmission delay, road-cloud transmission delay, environmental feature vector, AP index and target model parameters is iteratively performed until the target model parameters corresponding to each frame of collaborative perception result are generated, so as to obtain the target optimized vehicle-side perception model and target optimized road-side perception model corresponding to each frame of collaborative perception result based on the target model parameters corresponding to each frame of collaborative perception result.

[0105] Specifically, in the embodiment of the present application, the main computational delay of the target-level collaborative algorithm framework is in the point cloud target detection model at both ends of the vehicle and the road. The embodiment of the present application can set an optimization model here to dynamically optimize the point cloud target detection algorithm, such as Figure 7 As shown, the left side of the red line shows the specific operation process of the dynamic vehicle-road model for the optimization strategy. Figure 7 The right side of the red line shows the research process of the dynamic vehicle-road model on the dynamic optimization strategy.

[0106] Based on the above, the time process of vehicle-road-cloud collaborative perception is modeled as follows:

[0107] T c =max{(T fv +T v ), (T fr +T r )}+T ff (8)

[0108] It is understandable that the optimal algorithm flow is to assume that T fv , T v , T fr , T r and T ff When fixed, the algorithm parameters M with the best performance are selected as the best solution through the offline training process, resulting in T c is also fixed, in which case the algorithm is called f globalbest , which cannot dynamically adjust according to different perception and communication environments and may not be able to N The data input is completed before t′ N The perception result output. For this problem, algorithm f conservative When deploying, we tend to use the model parameter pair that consumes less time. The running comparison process of the two is as follows: Figure 8 shown.

[0109] In the actual implementation process, the embodiment of the present application can dynamically update the parameters M of the vehicle-side and roadside detection algorithms online at a time interval of Δt during the deployment process, so that the entire vehicle-road cooperative perception algorithm can adapt and dynamically adjust to the current perception situation, thereby achieving optimal performance. The process is f dynamic , as shown in the following pseudo code:

[0110]

[0111]

[0112] In order to formulate the above dynamic strategy, the embodiment of the present application needs to consider establishing a full representation of the current vehicle-road-cloud collaborative perception situation to determine how to design the conversion strategy of the embodiment of the present application. Specifically, the factors affecting the current vehicle-road-cloud collaborative perception situation mainly include the complexity of the perception environment. t , model parameter set (M v , M r ) and the target-level information transmission delay between the vehicle, road and cloud (T v , T r ).

[0113] In the specific implementation process, the embodiment of the present application can predict the given environment feature vector e t and transmission delay (T v , T r ) in different model parameter sets (M v , M r ) is actually to transform the problem of expressing the perception influencing factors into a regression problem. The embodiment of the present application trains a total of 14 Pointpillars models with the same structure but different scales, seven for each vehicle and road, and sets the optimal model parameters (M vbest , M rbest ), and then select the optimal model parameter set to be used in the future Δt time.

[0114] As a feasible method, the present embodiment trains a random forest model to predict the parameter (M vbest , M rbest ), and uses the mean square error as the loss function. The regression model training process is shown in the following pseudo code:

[0115]

[0116]

[0117] In addition, the embodiment of the present application needs to supplement the above pseudo code with the following content:

[0118] 1. Environment feature vector setting.

[0119] In order to fully improve the performance of the regression model and minimize the inference time, the embodiment of the present application performs feature engineering environment feature modeling on the traffic scene based on the lidar traffic data. These features include:

[0120] 1) num_bbox (number of bounding boxes);

[0121] 2) bounding box occlusion (bounding box occlusion);

[0122] 3) bounding box truncation situation;

[0123] 4) littlepoints objects (small targets);

[0124] 5) time of day (time).

[0125] 2. Parameter set settings for Pointpillars (PP) models of different scales.

[0126] The sizes of the backbone network and the neck network of the basic model, i.e., the size of PP3, are shown in Table 1:

[0127] Table 1

[0128]

[0129] In the embodiment of the present application, the size of the model can be adjusted based on the depth scaling factor α≈1.23, the width scaling factor β≈1.61 and the scaling φ. In the size adjustment, the size of the convolution layer of each stage of the model is adjusted, and the AP indicators of 3D and BEV of different models are calculated to reflect their performance, as shown in Table 2:

[0130] Table 2

[0131]

[0132]

[0133] In the embodiment of the present application, the model calculation delay is used as a priori input for algorithm execution model parameter search. In the inference process, it can be used as a priori experience to perform model parameter search. The embodiment of the present application collects the algorithm calculation delay of different parameter algorithm models in a large number of scenarios to obtain the statistical data of algorithm calculation delay, and converts the statistical time data t of different scale algorithms into average ,t best ,t worst Used as collaborative sensing dynamic optimization algorithm f dynamic The input is shown in Table 3:

[0134] Table 3

[0135]

[0136] It should be noted that the embodiments of the present application can use the point cloud target detection data part of the existing KITTI dataset as a large-scale pre-training dataset for the vehicle-road multi-scale Pointpillar target detection model, and use the vehicle-road point cloud target detection data part of the DAIR-V2X dataset as the main experimental verification and training dataset.

[0137] For the data set used by the regression model involved in the dynamic optimization strategy, the embodiment of the present application can adopt the frame-by-frame performance indicators generated after reasoning based on the trained multi-scale point cloud target detection model on the screened DAIR-V2X data set, and the environmental indicators generated by the above method to jointly form a new data set, and sort the time series to generate a training set and a validation set respectively.

[0138] The following is a quantitative verification of the PAP indicators and dynamic optimization strategies involved in the vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment of this application.

[0139] 1. Quantitative verification of PAP indicators

[0140] In the embodiment of the present application, three vehicle-road model pairs may be selected for experiment to prove that the performance of the model in the real-time perception scenario may be reduced compared with the offline test. The three vehicle-road model pairs may be (PP4, PP2), (PP6, PP4) and (PP7, PP6); the initial transmission delay T in the experiment v0 50ms, T r0 The AP and PAP values ​​of the selected vehicle-road model pairs are shown in Table 4.

[0141] Table 4

[0142]

[0143] As can be seen from Table 4, the module pairs with higher computational complexity also have a higher failure rate in the real-time environment, resulting in a significant decrease in real-time precision performance. Finally, the module pair (PP7, PP6) performed the worst among the three module pairs; at the same time, it shows that AP as a vector reflecting the offline accuracy of the model is significantly different from the PAP for real-time problems defined in the embodiments of this application, which further highlights that PAP is a comprehensive indicator that takes into account both real-time and accuracy.

[0144] 2. Quantitative verification of dynamic optimization strategy

[0145] In order to demonstrate the universality of the dynamic optimization strategy, the embodiment of the present application performs an initial transmission delay T v0 50ms, 75ms and 100ms, T r0The crossover experiment was conducted with the fusion delay ε of 20ms, 35ms and 50ms and the fusion delay ε of 100ms and 200ms. Table 5 shows the dynamic strategy algorithm f proposed in the embodiment of the present application. dynamic , the global optimal single algorithm f globalbest and the conservative method with time redundancy f conservative performance.

[0146] Table 5

[0147]

[0148]

[0149] In the conservative deployment algorithm, the embodiment of the present application may choose to leave 50% of the time redundancy or a conservative threshold t set manually. p = 20ms, so that it can be used after completing the vehicle and road perception processing, which means that the conservative strategy ensures that the output of each perception result can be used, that is, the failure rate is 0%; in terms of vehicle-cloud fusion, the embodiment of the present application can use the classic weighted Hungarian matching algorithm, and the calculation time of each frame is on the order of 10 -2 ms, so it can be ignored.

[0150] Taking the first set of settings as an example, for f globalbest In most cases, PP6, which ranks second in accuracy when deployed on the roadside, and PP4, which ranks fourth in accuracy when deployed on the vehicle side, can achieve the highest perception accuracy while meeting the perception output requirements. However, due to fluctuations in the perception environment and transmission delays, the results cannot be fused in time in some frames, resulting in lower perception accuracy in some frames. globalbest This performance is lost in these frames, thus providing room for perceptual optimization in these frames.

[0151] The dynamic algorithm in the embodiment of the present application is most of the time the same as the global optimal single algorithm f globalbest It remains consistent, but in the frames where the global optimal algorithm cannot output the perception results, it can output the perception results in real time by dynamically adjusting the model parameters, thereby improving the overall perception performance under the constraints of the perception output requirements; at the same time, compared with the conservatively deployed algorithm, the excess redundant time of the model deployment is fully utilized to achieve higher accuracy, which improves the dynamic perception performance while ensuring the output security.

[0152] After sufficient data verification, the algorithm f designed in the embodiment of the present application for dynamic adjustment of computing scenarios dynamic , and f globalbest There is a significant dynamic PAP improvement of about 5.8% compared to f conservativeThis is a 27.5% improvement compared to the original value, which strongly proves the necessity of dynamically adjusting the parameter M and f dynamic The effectiveness of the design.

[0153] According to the vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment proposed in an embodiment of the present application, multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in a target traffic scene are respectively obtained, and the multi-frame vehicle-end point cloud data and the multi-frame road-end point cloud data are respectively input into a preset vehicle-end perception model and a road-end perception model to generate a vehicle-end perception result and a road-end perception result; the vehicle-end perception result and the road-end perception result are fused on the target edge cloud to obtain a collaborative perception result and a fusion duration, and at least one target vehicle-end information node and at least one road-end information node are respectively obtained. The vehicle-cloud transmission delay and road-cloud transmission delay between the node and the target edge cloud are used to calculate the total fusion output time based on the vehicle-cloud transmission delay, road-cloud transmission delay and fusion time, and the real-time perception performance index corresponding to the collaborative perception result is calculated based on the total fusion output time and the preset perception output lag parameter; based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy, the vehicle-side perception model and the road-side perception model are subjected to preset dynamic optimization operations to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model, and the vehicle-road-cloud real-time collaborative perception result is generated based on the target optimized vehicle-side perception model and the target optimized road-side perception model. This application can improve the real-time and accurate performance of the target-level collaborative perception algorithm by dynamically adjusting the strategy.

[0154] Secondly, the vehicle-road-cloud real-time collaborative perception device for an intelligent connected environment proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0155] Fig. 9 It is a block diagram of a vehicle-road-cloud real-time collaborative perception device for an intelligent connected environment according to an embodiment of the present application.

[0156] like Fig. 9 As shown, the vehicle-road-cloud real-time collaborative perception device 10 for an intelligent connected environment includes: a perception module 100, an indicator construction module 200 and a dynamic optimization module 300.

[0157] Among them, the perception module 100 is used to respectively obtain multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in the target traffic scene, and input the multi-frame vehicle-end point cloud data and the multi-frame road-end point cloud data into the preset vehicle-end perception model and road-end perception model respectively to generate vehicle-end perception results and road-end perception results.

[0158] The indicator construction module 200 is used to fuse the vehicle-side perception results and the road-side perception results through the target edge cloud to obtain the collaborative perception results and fusion time, and respectively obtain the vehicle-cloud transmission delay and road-cloud transmission delay between at least one target vehicle-side information node and at least one road-side information node and the target edge cloud, so as to calculate the total fusion output time based on the vehicle-cloud transmission delay, the road-cloud transmission delay and the fusion time, and calculate the real-time perception performance indicator corresponding to the collaborative perception result according to the total fusion output time and the preset perception output lag parameter.

[0159] The dynamic optimization module 300 is used to perform preset dynamic optimization operations on the vehicle-side perception model and the road-side perception model based on real-time perception performance indicators and preset real-time dynamic algorithm optimization strategies to obtain a target optimized vehicle-side perception model and a target optimized road-side perception model, and generate a vehicle-road-cloud real-time collaborative perception result based on the target optimized vehicle-side perception model and the target optimized road-side perception model.

[0160] Optionally, in one embodiment of the present application, the perception module 100 includes: an acquisition unit and a feature extraction unit.

[0161] Among them, the acquisition unit is used to respectively obtain multiple frames of vehicle-end point cloud data and multiple frames of road-end point cloud data through at least one target vehicle-end information node and at least one road-end information node, and use the vehicle-end perception model to detect and track the multiple frames of vehicle-end point cloud data to generate a vehicle-end perception result.

[0162] The feature extraction unit is used to perform road-side target detection and feature extraction operations on multiple frames of road-side point cloud data using a road-side perception model to generate a road-side perception result.

[0163] Optionally, in one embodiment of the present application, the indicator construction module 200 includes: a fusion unit, used to perform coordinate conversion processing on the vehicle-side perception results and the road-side perception results through the target edge cloud, and perform target-level information fusion detection operations on the vehicle-side perception results and the road-side perception results after the coordinate conversion processing to generate collaborative perception results and fusion duration.

[0164] Optionally, in one embodiment of the present application, the indicator construction module 200 further includes: a first determination unit, a first calculation unit, a comparison unit, a second calculation unit, a third calculation unit and a fourth calculation unit.

[0165] Among them, the first determination unit is used to determine the perception time of the roadside model corresponding to the vehicle-side perception model and the perception time of the vehicle-side model corresponding to the road-side perception model.

[0166] The first calculation unit is used to calculate the total fusion output time according to the road-side model perception time, the vehicle-side model perception time, the vehicle-cloud transmission delay, the road-cloud transmission delay and the fusion time.

[0167] The comparison unit is used to compare the total duration of the fusion output and the hysteresis parameter of the perception output.

[0168] The second calculation unit is used to calculate the first perception result performance corresponding to each frame of vehicle endpoint cloud data in multiple frames of vehicle endpoint cloud data when the total duration of the fusion output is less than the perception output lag parameter.

[0169] The third calculation unit is used to calculate the second perception result performance corresponding to each frame of vehicle endpoint cloud data in multiple frames of vehicle endpoint cloud data when the total duration of the fusion output is greater than or equal to the perception output lag parameter.

[0170] The fourth calculation unit is used to obtain the total number of frames of multi-frame vehicle endpoint cloud data, and calculate the sum of real-time perception performance corresponding to the multi-frame vehicle endpoint cloud data based on the first perception result performance and the second perception result performance, and calculate the real-time perception performance index based on the sum of real-time perception performance and the total number of frames.

[0171] Optionally, in one embodiment of the present application, the dynamic optimization module 300 includes: a modeling unit, a second determination unit, a first generation unit, a second generation unit and an iteration unit.

[0172] Among them, the modeling unit is used to perform feature engineering environmental feature modeling on the target traffic scene corresponding to each frame of collaborative perception results in the multi-frame collaborative perception results, so as to obtain the environmental feature vector corresponding to the target traffic scene.

[0173] The second determination unit is used to construct multiple groups of model parameter sets of different scales for the vehicle-side perception model and the road-side perception model respectively, and determine the AP index corresponding to each group of model parameter sets of different scales in the multiple groups of model parameter sets of different scales.

[0174] The first generation unit is used to input the vehicle-cloud transmission delay, road-cloud transmission delay, environmental feature vector and AP index pair corresponding to the first frame of collaborative perception results in the multi-frame collaborative perception results into a pre-trained regression prediction model to generate multiple groups of model parameters of different scales to meet the target model parameters required by preset real-time perception performance indicators.

[0175] The second generation unit is used to update the vehicle-cloud transmission delay, road-cloud transmission delay, environmental feature vector and AP index according to the vehicle-side perception model and the road-side perception model corresponding to the target model parameters after a preset time interval, so as to obtain a new vehicle-cloud transmission delay, a new road-cloud transmission delay, a new environmental feature vector and a new AP index corresponding to the second frame collaborative perception result, and input the new vehicle-cloud transmission delay, the new road-cloud transmission delay, the new environmental feature vector, the new AP index and the target model parameters into the regression prediction model to generate multiple groups of model parameters of different scales that meet the new target model parameters required by the preset real-time perception performance indicators.

[0176] The iteration unit is used to iteratively execute the update operations of vehicle-cloud transmission delay, road-cloud transmission delay, environmental feature vector, AP index and target model parameters until the target model parameters corresponding to each frame of collaborative perception result are generated, so as to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model corresponding to each frame of collaborative perception result based on the target model parameters corresponding to each frame of collaborative perception result.

[0177] It should be noted that the aforementioned explanation of the embodiment of the vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment is also applicable to the vehicle-road-cloud real-time collaborative perception device for an intelligent connected environment of this embodiment, and will not be repeated here.

[0178] The vehicle-road-cloud real-time collaborative perception device for an intelligent connected environment proposed in an embodiment of the present application includes a perception module for respectively obtaining multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in a target traffic scene, and inputting the multi-frame vehicle-end point cloud data and the multi-frame road-end point cloud data into a preset vehicle-end perception model and a road-end perception model, respectively, to generate a vehicle-end perception result and a road-end perception result; an indicator construction module for fusing the vehicle-end perception result and the road-end perception result through a target edge cloud to obtain a collaborative perception result and a fusion duration, and respectively obtaining at least one target vehicle-end information node and at least one road-end information node. The vehicle-cloud transmission delay and road-cloud transmission delay between the end information node and the target edge cloud are used to calculate the total fusion output time based on the vehicle-cloud transmission delay, road-cloud transmission delay and fusion time, and the real-time perception performance index corresponding to the collaborative perception result is calculated based on the total fusion output time and the preset perception output lag parameter; the dynamic optimization module is used to perform preset dynamic optimization operations on the vehicle-side perception model and the road-side perception model based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model, and generate the vehicle-road-cloud real-time collaborative perception result based on the target optimized vehicle-side perception model and the target optimized road-side perception model. This application can improve the real-time and accurate performance of the target-level collaborative perception algorithm by dynamically adjusting the strategy.

[0179] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0180] A memory 1001 , a processor 1002 , and a computer program stored in the memory 1001 and executable on the processor 1002 .

[0181] When the processor 1002 executes the program, the vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment provided in the above-mentioned embodiment is implemented.

[0182] Furthermore, the electronic device further comprises:

[0183] The communication interface 1003 is used for communication between the memory 1001 and the processor 1002 .

[0184] The memory 1001 is used to store computer programs that can be executed on the processor 1002 .

[0185] The memory 1001 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0186] If the memory 1001, the processor 1002 and the communication interface 1003 are implemented independently, the communication interface 1003, the memory 1001 and the processor 1002 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0187] Optionally, in a specific implementation, if the memory 1001, the processor 1002 and the communication interface 1003 are integrated on a chip, the memory 1001, the processor 1002 and the communication interface 1003 can communicate with each other through an internal interface.

[0188] The processor 1002 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0189] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment is implemented.

[0190] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment.

[0191] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0192] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0193] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0194] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0195] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0196] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0197] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0198] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment, characterized in that: The following steps are involved: Respectively obtain multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in the target traffic scene, and input the multi-frame vehicle-end point cloud data and the multi-frame road-end point cloud data into a preset vehicle-end perception model and a road-end perception model, respectively, to generate a vehicle-end perception result and a road-end perception result; The vehicle-side perception result and the road-side perception result are fused through the target edge cloud to obtain a collaborative perception result and a fusion duration, and the vehicle-cloud transmission delay and the road-cloud transmission delay between the at least one target vehicle-side information node and the at least one road-side information node and the target edge cloud are respectively obtained, so as to calculate the total fusion output duration based on the vehicle-cloud transmission delay, the road-cloud transmission delay and the fusion duration, and calculate the real-time perception performance index corresponding to the collaborative perception result according to the total fusion output duration and a preset perception output lag parameter; Based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy, the vehicle-side perception model and the road-side perception model are subjected to preset dynamic optimization operations to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model, and the vehicle-road-cloud real-time collaborative perception results are generated according to the target optimized vehicle-side perception model and the target optimized road-side perception model.

2. The method according to claim 1, characterized in that The method of respectively acquiring multiple frames of vehicle-end point cloud data and multiple frames of road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in the target traffic scene, and inputting the multiple frames of vehicle-end point cloud data and the multiple frames of road-end point cloud data into a preset vehicle-end perception model and a road-end perception model, respectively, to generate a vehicle-end perception result and a road-end perception result, includes: Acquire multiple frames of vehicle endpoint cloud data and multiple frames of road endpoint cloud data respectively through the at least one target vehicle-end information node and the at least one road-end information node, and detect and track the multiple frames of vehicle endpoint cloud data using the vehicle-end perception model to generate the vehicle-end perception result; The road-side perception model is used to perform road-side target detection and feature extraction operations on the multi-frame road-side point cloud data to generate the road-side perception result.

3. The method according to claim 2, characterized in that The fusing the vehicle-side perception result and the road-side perception result through the target edge cloud to obtain a collaborative perception result and a fusion duration includes: The target edge cloud performs coordinate conversion processing on the vehicle-side perception results and the road-side perception results, and performs target-level information fusion detection operation on the vehicle-side perception results and the road-side perception results after the coordinate conversion processing to generate the collaborative perception results and the fusion duration.

4. The method according to claim 3, characterized in that The step of respectively obtaining the vehicle-to-cloud transmission delay and the road-to-cloud transmission delay between the at least one target vehicle-side information node, the at least one road-side information node, and the target edge cloud, calculating the total fusion output duration based on the vehicle-to-cloud transmission delay, the road-to-cloud transmission delay, and the fusion duration, and calculating the real-time perception performance index corresponding to the collaborative perception result according to the total fusion output duration and a preset perception output lag parameter, includes: Determine the roadside model perception duration corresponding to the vehicle-side perception model and the vehicle-side model perception duration corresponding to the roadside perception model; Calculate the total fusion output duration according to the roadside model perception duration, the vehicle-side model perception duration, the vehicle-cloud transmission delay, the road-cloud transmission delay and the fusion duration; comparing the total duration of the fusion output and the perception output lag parameter; When the total duration of the fusion output is less than the perception output lag parameter, calculating the first perception result performance corresponding to each frame of vehicle endpoint cloud data in the multiple frames of vehicle endpoint cloud data; When the total duration of the fusion output is greater than or equal to the perception output lag parameter, calculating the second perception result performance corresponding to each frame of vehicle endpoint cloud data in the multiple frames of vehicle endpoint cloud data; Obtain the total number of frames of the multi-frame vehicle endpoint cloud data, and calculate the sum of real-time perception performance corresponding to the multi-frame vehicle endpoint cloud data according to the first perception result performance and the second perception result performance, and calculate the real-time perception performance index based on the sum of real-time perception performance and the total number of frames.

5. The method according to claim 1, characterized in that The method of performing a preset dynamic optimization operation on the vehicle-side perception model and the road-side perception model based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy to obtain a target optimized vehicle-side perception model and a target optimized road-side perception model includes: Performing feature engineering environment feature modeling on the target traffic scene corresponding to each frame of collaborative perception results in the multi-frame collaborative perception results to obtain an environment feature vector corresponding to the target traffic scene; Constructing a plurality of sets of model parameter sets of different scales for the vehicle-side perception model and the road-side perception model respectively, and determining an AP index corresponding to each set of model parameter sets of different scales in the plurality of sets of model parameter sets of different scales; Inputting the vehicle-cloud transmission delay, the road-cloud transmission delay, the environmental feature vector and the AP index pair corresponding to the first frame of the collaborative sensing results in the multi-frame collaborative sensing results into a pre-trained regression prediction model to generate target model parameters that meet the preset real-time sensing performance index requirements in the multiple groups of model parameter sets of different scales; After a preset time interval, the vehicle-to-cloud transmission delay, the road-to-cloud transmission delay, the environmental feature vector and the AP index are updated according to the vehicle-side perception model and the road-side perception model corresponding to the target model parameters to obtain a new vehicle-to-cloud transmission delay, a new road-to-cloud transmission delay, a new environmental feature vector and a new AP index corresponding to the second frame collaborative perception result, and the new vehicle-to-cloud transmission delay, the new road-to-cloud transmission delay, the new environmental feature vector, the new AP index and the target model parameters are input into the regression prediction model to generate a new target model parameter that meets the preset real-time perception performance index requirements among the multiple groups of model parameters of different scales; Iteratively execute the update operations of the vehicle-cloud transmission delay, the road-cloud transmission delay, the environmental feature vector, the AP index and the target model parameters until the target model parameters corresponding to the collaborative perception results of each frame are generated, so as to obtain the target optimized vehicle-side perception model and the target optimized road-side perception model corresponding to the collaborative perception results of each frame based on the target model parameters corresponding to the collaborative perception results of each frame.

6. A vehicle-road-cloud real-time collaborative perception device for an intelligent network environment, characterized in that: include: A perception module, used to respectively obtain multi-frame vehicle-end point cloud data and multi-frame road-end point cloud data corresponding to at least one target vehicle-end information node and at least one road-end information node in a target traffic scene, and input the multi-frame vehicle-end point cloud data and the multi-frame road-end point cloud data into a preset vehicle-end perception model and a road-end perception model, respectively, to generate a vehicle-end perception result and a road-end perception result; An indicator construction module is used to fuse the vehicle-side perception result and the road-side perception result through the target edge cloud to obtain a collaborative perception result and a fusion duration, and respectively obtain the vehicle-cloud transmission delay and the road-cloud transmission delay between the at least one target vehicle-side information node and the at least one road-side information node and the target edge cloud, so as to calculate the total fusion output duration based on the vehicle-cloud transmission delay, the road-cloud transmission delay and the fusion duration, and calculate the real-time perception performance indicator corresponding to the collaborative perception result according to the total fusion output duration and a preset perception output lag parameter; A dynamic optimization module is used to perform preset dynamic optimization operations on the vehicle-side perception model and the road-side perception model based on the real-time perception performance index and the preset real-time dynamic algorithm optimization strategy to obtain a target optimized vehicle-side perception model and a target optimized road-side perception model, and generate a vehicle-road-cloud real-time collaborative perception result based on the target optimized vehicle-side perception model and the target optimized road-side perception model.

7. The device according to claim 6, characterized in that The perception module comprises: an acquisition unit, configured to respectively acquire multiple frames of vehicle-end point cloud data and multiple frames of road-end point cloud data through the at least one target vehicle-end information node and the at least one road-end information node, and detect and track the multiple frames of vehicle-end point cloud data using the vehicle-end perception model to generate the vehicle-end perception result; A feature extraction unit is used to perform road-end target detection and feature extraction operations on the multi-frame road-end point cloud data using the road-end perception model to generate the road-end perception result.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment as described in any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the vehicle-road-cloud real-time collaborative perception method for an intelligent connected environment as described in any one of claims 1-5.

Images