Vehicle and cloud data fusion method and device

Abstract

This application discloses a method and apparatus for vehicle-cloud data fusion, belonging to the field of autonomous driving technology. The method includes: receiving data transmitted from the cloud, the cloud-transmitted data including driving data of a cloud-perceived target; acquiring driving data of a vehicle-observed target; matching the cloud-perceived target with the vehicle-observed target; and, if the matching is successful, fusing the driving data of the cloud-perceived target and the vehicle-observed target to obtain a predicted state sequence of the vehicle-observed target. This method matches and fuses the cloud-transmitted target with the target perceived by the autonomous vehicle, obtaining more reliable target information, thereby providing sufficient support and effective guidance for downstream regulatory control.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving technology, and in particular relates to a method and device for vehicle-cloud data fusion. Background Technology

[0002] Cloud-to-vehicle technology enables communication between a cloud computing platform and vehicles, allowing for the uploading and downloading of vehicle data. This enables autonomous driving devices to acquire key information such as the real-time location, speed, heading angle, and local path of other vehicles, enhancing their predictive and decision-making capabilities in complex traffic environments.

[0003] In existing technologies, cloud-based data delivery often lacks effective strategic utilization and is generally used as a safety redundancy or fallback measure, which prevents the full realization of the data's potential value and limits the performance improvement of autonomous driving systems in decision-making, planning, and control. Summary of the Invention

[0004] This application provides a method and apparatus for vehicle-cloud data fusion, which can effectively match and fuse cloud data with vehicle data, thereby improving the planning capabilities of autonomous driving systems.

[0005] In a first aspect, embodiments of this application provide a method for vehicle-cloud data fusion, the method comprising: Receive data sent from the cloud, which includes driving data of targets sensed by the cloud; Acquire driving data of the target observed by the vehicle. The driving data of the target observed by the vehicle is collected by sensors deployed on the autonomous driving equipment. Match the targets perceived in the cloud with the targets observed on the vehicle; If a match is successful, the driving data of the cloud-sensed target and the driving data of the vehicle-observed target are merged to obtain the predicted state sequence of the vehicle-observed target.

[0006] Secondly, embodiments of this application provide a vehicle-cloud data fusion device, the device comprising: The receiving module is used to receive data sent from the cloud, which includes driving data of the cloud-sensed target.

[0007] The acquisition module is used to acquire driving data of the vehicle-mounted observation target, which is collected by sensors deployed on the autonomous driving equipment.

[0008] A matching module is used to match the cloud-sensing target with the vehicle-mounted observation target; The fusion module is used to fuse the driving data of the cloud-sensing target and the driving data of the vehicle-mounted observation target when the matching is successful, so as to obtain the predicted state sequence of the vehicle-mounted observation target.

[0009] Thirdly, embodiments of this application provide a computer device, the device including: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method of any one of the first aspects above.

[0010] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the method of any one of the first aspects described above.

[0011] Fifthly, embodiments of this application provide a computer program product comprising a computer program that, when executed by a processor, implements the method of any one of the first aspects described above.

[0012] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects: The vehicle-cloud data fusion method provided in this application involves associating and matching driving targets perceived in the cloud with those observed by vehicle sensors. Upon successful matching, the two types of data are fused to generate a predicted state sequence of the vehicle-observed targets. This method effectively leverages data distributed from the cloud, achieving efficient fusion of vehicle and cloud information. Furthermore, it transmits more accurate and stable target information downstream to the control system, thereby improving the accuracy and reliability of the autonomous driving system in path planning, driving decisions, and control execution. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below.

[0014] Figure 1 This is a flowchart illustrating a vehicle-cloud data fusion method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the transformation relationship between a spatial coordinate system and a vehicle coordinate system, provided in an embodiment of this application. Figure 3 This is a flowchart illustrating another vehicle-cloud data fusion method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a vehicle-cloud data fusion device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0016] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples consistent with some aspects of this application as detailed in the appended claims.

[0017] Cloud-to-Vehicle (C2V) technology refers to a technology that enables communication between vehicles and cloud computing platforms to transmit and process data. In this technology, vehicles can upload collected data to the cloud and simultaneously receive analyzed and processed information from the cloud, such as the status of other vehicles, traffic data, navigation updates, and software updates. Therefore, this technology is of great significance to autonomous driving devices. Autonomous driving devices can obtain key information about other vehicles stored in the cloud, such as real-time location, speed, heading angle, and planned local paths, essentially providing the autonomous driving device with real-time path planning information from other vehicles, greatly improving its predictive and decision-making capabilities in complex traffic environments. Furthermore, the application of this technology allows autonomous driving devices to participate more intelligently in traffic flow. Through information transmitted from the cloud, autonomous driving devices can anticipate the driving intentions of other vehicles, thereby making more rational driving decisions, reducing traffic accidents, improving road efficiency, or avoiding congested areas, further optimizing the travel experience.

[0018] Currently, the utilization of cloud-deployed data is generally to output it separately as redundant information to downstream planning and control modules to provide security redundancy. However, in this approach, planning and control modules often only use cloud-deployed information as a fallback measure, lacking further strategic utilization, resulting in insufficient and inflexible use of the information.

[0019] This shows that the application of cloud-based data in existing autonomous driving technologies is still insufficient, making it difficult to balance real-time performance, accuracy, and effective utilization.

[0020] To address this technical problem, embodiments of this application provide a method and apparatus for vehicle-cloud data fusion. In this method, by matching and fusing the operational targets transmitted from the cloud with the operational targets perceived by the autonomous vehicle, more reliable target information is obtained, thereby providing sufficient support and effective guidance for downstream regulatory control.

[0021] The vehicle-cloud data fusion method provided in this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] It should be understood that the implementing entity of the solution in this application can be the processor of an autonomous driving device, such as an in-vehicle infotainment system.

[0023] Figure 1 A flowchart illustrating a vehicle-cloud data fusion method provided in one embodiment of this application is shown.

[0024] like Figure 1 As shown, the method may include the following steps.

[0025] S110: Receives data sent from the cloud.

[0026] Cloud-based data delivery refers to data sent from a cloud server to an autonomous driving device via a communication link; this can be referred to as C2V data.

[0027] For example, the data delivered from the cloud includes driving data of at least one cloud-sensing target.

[0028] Cloud-based perceived targets refer to driving targets related to autonomous driving equipment. These can be other vehicles around the autonomous driving equipment, traffic participants, or road traffic facilities.

[0029] The driving data may include, but is not limited to, information such as the real-time position, speed, heading angle, acceleration, and planned local path of the target sensed in the cloud.

[0030] It should be understood that the source of the driving data of the cloud-sensing target can be the operation data collected and reported to the cloud by other vehicles, or the data collected by roadside detection equipment (such as roadside units, surveillance cameras, etc.). This application does not limit the specific source of the data.

[0031] S120: Acquire driving data of the target observed on the vehicle.

[0032] Vehicle-side observed targets refer to the driving targets that autonomous driving equipment observes on its own, which can be other vehicles, traffic participants, or other objects in the vicinity.

[0033] The driving data may also include, but is not limited to, information such as the real-time position, speed, heading angle, acceleration, and planned local path of the target observed by the vehicle.

[0034] It should be understood that the driving data of the target observed by the vehicle is collected by various sensors deployed on the autonomous driving equipment. The types of sensors may include, but are not limited to, LiDAR, cameras, millimeter-wave radar, etc.

[0035] In one implementation, the vehicle processor can process multi-source sensor data using on-vehicle perception algorithms (such as multi-sensor fusion algorithms). These multi-sensor fusion algorithms can be categorized into three types: data-level fusion, feature-level fusion, and target-level fusion. For example, in practical applications, the on-vehicle perception algorithm can combine feature-level fusion with target-level fusion to obtain more accurate target perception results. The target perception information obtained in this way can include, but is not limited to, lateral and longitudinal position, lateral and longitudinal velocity, acceleration, heading angle, and yaw rate, thus providing reliable input for subsequent data matching and fusion.

[0036] S130, Match the cloud-sensed target with the vehicle-mounted observed target.

[0037] By matching cloud-sensing targets with vehicle-side observation targets, the correspondence between cloud-based sensing targets and vehicle-side self-observation targets is determined.

[0038] In one implementation, cloud-sensing targets and vehicle-observed targets can be matched based on distance. Specifically, the distance difference between the cloud-sensing target and the vehicle-observed target, such as position or velocity, is calculated. If the difference is less than a preset threshold, the two are considered to correspond to the same target.

[0039] In one implementation, a similarity-based single-target matching method matches cloud-sensed targets with vehicle-observed targets. Specifically, by calculating the similarity metric between targets, the target pair with the highest similarity is selected for matching. For example, the similarity-based single-target matching method could be a nearest neighbor algorithm, a greedy nearest neighbor algorithm, or a gated nearest neighbor algorithm.

[0040] In one implementation, the matching of cloud-sensing targets and vehicle-side observation targets can be based on a globally optimal single hypothesis method. Specifically, candidate target pairs are constructed, and a global optimization matching is performed on all candidate target pairs to obtain a globally optimal matching result. Examples of globally optimal single hypothesis methods include the Hungarian algorithm, auction algorithms, etc.

[0041] In one implementation, matching cloud-sensing targets with vehicle-mounted observation targets can be based on probabilistic data association. Specifically, by calculating the association probability between cloud-sensing targets and vehicle-mounted observation targets, the matching relationship is determined based on the probability magnitude, thereby improving the robustness of matching in complex scenarios.

[0042] S140. If the matching is successful, the driving data of the cloud-sensed target and the driving data of the vehicle-side observed target are merged to obtain the predicted state sequence of the vehicle-side observed target.

[0043] It should be understood that data from vehicle-mounted observation targets typically have higher real-time performance and accuracy. Therefore, during the fusion process, the predicted state sequence is generated based on the vehicle-mounted observation targets.

[0044] The predicted state sequence of a target observed from the vehicle refers to the sequence of target state changes over time after being corrected based on vehicle-mounted observation data and cloud-based sensing data. It is used to reflect the target's motion trend and dynamic characteristics in the future.

[0045] In one implementation, the driving data of the vehicle-mounted observed target is used as a basis, and the driving data of the cloud-sensed target is integrated. The predicted state results of the vehicle-mounted observed target are updated using a filtering algorithm to generate a predicted state sequence of the vehicle-mounted observed target.

[0046] For example, the Kalman filter algorithm can be invoked to fuse the target states.

[0047] In this implementation, the driving data of the target observed by the vehicle is fused as a basis, and the driving data of the target perceived by the cloud is updated by combining the filtering algorithm. While ensuring the real-time and continuous nature of the vehicle data, the uncertainty caused by delay and noise in the cloud data is effectively suppressed, making the fusion result smoother, more stable and closer to the true state of the target, thereby improving the accuracy and reliability of the predicted state sequence.

[0048] The vehicle-cloud data fusion method provided in this embodiment associates and matches driving targets perceived in the cloud with those observed by vehicle sensors. Upon successful matching, the two types of data are fused to generate a predicted state sequence of the vehicle-observed targets. This method fully utilizes data transmitted from the cloud, achieving effective fusion of vehicle and cloud information. Furthermore, it transmits more accurate and stable target information downstream to the control system, thereby improving the accuracy and reliability of the autonomous driving system in path planning, driving decisions, and control execution.

[0049] In some embodiments, the present application may further include S150.

[0050] S150. In the event of a failed match, each failed cloud-sensing target is included in the tracking process as a virtual target until the preset life threshold of the tracked cloud-sensing target is reached.

[0051] For example, a virtual bounding box can be constructed for each unmatched cloud-sensed target, and a unique identifier can be assigned to it. This target is then included in the tracking target list, and a life threshold is set. Within this life threshold, the virtual target can participate in subsequent processing as a potentially valid target, thereby providing information on distant or occluded targets for planning and control. Based on this information, the planning and control module can choose a conservative strategy for decision-making.

[0052] The method in this embodiment can make full use of the data sent from the cloud even when matching fails, avoiding the problem of target loss caused by short-term communication delays or fluctuations in vehicle-side perception, and ensuring that cloud information continues to participate in subsequent data fusion and decision-making within a reasonable time frame.

[0053] In some embodiments, matching cloud-sensing targets with vehicle-mounted observation targets may include steps S131-S134.

[0054] S131. Pre-match the cloud-sensing targets and the vehicle-side observation targets to obtain multiple sets of candidate target pairs.

[0055] For example, pre-matching can be done by filtering based on spatial or motion features to create a candidate set of potentially matching target pairs. This step can reduce the computational cost of subsequent global matching.

[0056] Each candidate target pair may include at least one cloud-sensing target and at least one vehicle-mounted observation target.

[0057] S132. For each pair of candidate targets, calculate the matching cost based on the spatial position and motion differences between the cloud-sensed target and the vehicle-observed target.

[0058] For example, the spatial position difference feature refers to the relative distance between targets in two-dimensional or three-dimensional space, and may include at least one of Mahalanobis distance and position difference. It should be understood that the position difference here refers to Euclidean position difference.

[0059] For example, the motion difference feature refers to the difference in the motion state of the target, such as speed and heading, and may include at least one of speed difference and heading angle difference.

[0060] In other words, the matching cost can be calculated based on Mahalanobis distance and / or position difference, as well as velocity difference and / or heading angle difference, thereby quantifying the credibility of candidate targets. The lower the matching cost, the more likely the candidate target pair represents the same target.

[0061] Therefore, in one implementation, before performing step S132, the method of this application further includes the following steps.

[0062] For each pair of candidate targets, the Mahalanobis distance is calculated based on the observation state vector and covariance matrix of the target observed from the vehicle and the perception state vector of the target perceived from the cloud.

[0063] The position difference is calculated based on the geometric straight-line distance between each pair of candidate targets in a unified coordinate system.

[0064] Calculate the overall difference in velocity vectors for each group of candidate targets, and then calculate the velocity difference.

[0065] Compare the differences in motion direction angles between each pair of candidate targets; for example, normalize the motion direction angles of both pairs and then calculate the heading angle difference. Assume the autonomous vehicle obtains the set of targets observed from the vehicle. (N is the number of targets observed by the vehicle), and the set of sensing targets sent from the cloud. (M is the number of perceived targets), unify the two types of targets to the same coordinate system, and extrapolate to the same timestamp.

[0066] For any target Its state vector is defined as: (1) Where x and y represent the two-dimensional positions of the target in the same coordinate system; This represents the velocity components of the target in the same coordinate system; This represents the target's heading angle. This state vector can describe the target's complete dynamic information at a specific moment.

[0067] For any candidate pair (i, j), calculate the Mahalanobis distance to measure the difference in position between the vehicle-observed target and the cloud-sensed target, as shown in the following formula: (2) in, For cloud-based target observation, , These represent the predicted state and prediction covariance of the target perceived by the vehicle.

[0068] in, This represents the observation position vector of the target perceived in the cloud. This represents the predicted position vector of the target perceived by the vehicle. This represents the covariance matrix of the vehicle-sensed target location prediction.

[0069] Mahalanobis distance takes into account the uncertainty of the predicted position of the target at the vehicle end, and can more accurately measure the reliability of the target matching.

[0070] In some embodiments, to save computational resources and improve matching efficiency, before performing step S132 and after calculating the Mahalanobis distance and position difference, the method of this application further includes: For each pair of candidate targets, if the Mahalanobis distance exceeds a preset threshold and / or the position difference exceeds a preset distance threshold, the pair of candidate targets is removed. That is, the pair of candidate targets whose Mahalanobis distance exceeds a preset threshold and / or whose position difference exceeds a preset distance threshold is removed.

[0071] A preset threshold value is used to limit the reasonable matching range of candidate target pairs in a statistical sense. It can be set according to actual needs, but this application does not limit it.

[0072] For example, the preset threshold value can be set using a chi-square distribution gate field to ensure that matches that are not probabilistically valid at a given confidence level are eliminated.

[0073] A preset distance threshold is used to limit the maximum permissible deviation of the candidate target in spatial position. It can be set according to actual needs, and this application does not limit it.

[0074] In a corresponding example, if the Mahalanobis distance of a candidate target pair is greater than the chi-square gate domain, and / or the L2 norm of their position difference is greater than a preset distance threshold, then the candidate target pair is directly determined to be uncorrelated and is removed.

[0075] The method in this embodiment introduces a Mahalanobis distance and position difference determination mechanism based on the observed state vector, covariance matrix and perceived state vector before calculating the matching cost. This can screen out unreasonable or unreliable matching pairs in the candidate target pairs in advance, reduce invalid calculations, and prevent abnormal data from entering the subsequent matching process, thereby improving the overall matching efficiency and accuracy.

[0076] S133. Construct a matching cost model based on the matching costs of multiple candidate target pairs.

[0077] The matching costs of each candidate target pair calculated in the previous step are organized to construct a matching cost model.

[0078] For example, the matching cost model can be represented as a cost matrix, where each row corresponds to the vehicle-side observed target and each column corresponds to the cloud-based perceived target.

[0079] For candidate target pairs, the cost formula is constructed as follows: (3) in, This represents the matching cost between the i-th vehicle-side observed target and the j-th cloud-based perceived target; This represents the cost calculated by combining spatial location differences and motion differences. For vehicle-mounted observation of target bounding box Cloud-based target bounding box The intersection and union ratio; β is the shape consistency weight.

[0080] In a possible implementation, the cost matrix is ​​obtained by calculating the cost matrix for each combination of vehicle-side and cloud-side objectives using the cost formula.

[0081] In addition, to handle missed or false alarm targets, corresponding penalty units can be added to the cost matrix to keep the matrix in square form, which facilitates subsequent global optimization solutions.

[0082] The matching cost model constructed in this way can comprehensively reflect the consistency of candidate target pairs in terms of spatial location, motion state, and shape features.

[0083] S134. The matching cost model is solved globally using an optimized matching algorithm to obtain the matching target pair.

[0084] In one implementation, the matching cost model is a cost matrix. The Hungarian matching algorithm is used to solve the cost matrix globally. From multiple sets of candidate target pairs, the candidate target pairs that satisfy the minimum global matching cost are selected and determined as the matching target pairs.

[0085] The Hungarian matching algorithm is called to perform global matching of the cost matrix. Under the condition of satisfying the one-to-one matching constraint between the vehicle-side observation target and the cloud-sensing target, the target matching combination with the minimum global matching cost is searched. Each candidate target pair in the target matching combination is determined as the matching target pair.

[0086] For example, when constructing the cost matrix, candidate target pairs with matching costs exceeding a preset threshold are eliminated to ensure the reliability and accuracy of the matching results.

[0087] This implementation constructs the matching cost model as a cost matrix and uses the Hungarian matching algorithm to solve it globally. This approach can obtain the overall optimal matching relationship among all candidate target pairs, avoid the accumulation of errors caused by local matching, and improve the global consistency and accuracy of matching.

[0088] The embodiments corresponding to steps S131-S134 above introduce pre-matching screening, joint modeling of spatial location differences and motion differences, and global solution based on optimization algorithms in the matching process. Compared with the traditional matching method that only relies on a single feature or local rules, it can improve the robustness and accuracy of matching while ensuring computational efficiency, significantly reduce the occurrence of mismatches and missed matches, and make the fusion results of cloud data and vehicle data more reliable.

[0089] In some embodiments, the method further includes preprocessing the data sent from the cloud before performing step S130.

[0090] For example, preprocessing may include anomaly detection, upsampling, and spatial transformation. Through preprocessing, data delivered from the cloud can be transformed into robust cloud-sensing targets that are spatiotemporally aligned with the vehicle-mounted observed targets.

[0091] In some embodiments, the anomaly detection process may include: performing anomaly detection on the driving data of the cloud-sensed target and filtering out driving data that meets preset anomaly conditions.

[0092] Anomaly detection is used to detect and filter out meaningless values, excessive delays, and duplicate data. Therefore, the preset anomaly conditions may include at least one of the following: The driving speed exceeds the preset speed threshold; The sampling delay exceeds the preset delay threshold; Data was repeatedly sent beyond the preset frame count threshold.

[0093] It should be understood that the preset speed threshold, preset delay threshold, and preset frame rate threshold can be set according to actual needs, and are not limited here.

[0094] Anomaly detection can eliminate abnormal or unreliable information, reduce erroneous matching and fusion interference, and improve the accuracy and stability of matching and fusion results.

[0095] Considering that data sent from the cloud usually has a delay and its frame rate is lower than the required frame rate of data sent from the vehicle, in order to ensure the matching accuracy with the observed targets on the vehicle, it is necessary to upsample the data sent from the cloud. That is, to predict and extrapolate the data at subsequent times based on the cloud-sensing target data sent at a certain time.

[0096] Therefore, in some embodiments, the step of upsampling the data sent from the cloud is as follows: Based on the historical driving data of the cloud-sensing target received, and with the sampling time of the driving data of the vehicle-side observed target as the target, the driving data of the cloud-sensing target is extrapolated and predicted using a motion model.

[0097] For example, the motion models include a constant acceleration motion model and a constant yaw rate motion model.

[0098] Assume the timestamp of the current frame (the i-th frame) is t. i The timestamp for data delivery from the cloud is t. i-n , where n represents the number of frames. t i-n At any given moment, the target's horizontal and vertical position, velocity, acceleration, heading angle, and yaw rate perceived in the cloud are x i-n y i-n a i-n v i-n θ i-n ω i-n , then t i The corresponding predicted value at time t can be obtained through the following extrapolation formula: (4) Where Δt=t i -t i-n F is the state transition matrix generated by the constant acceleration motion model and the constant yaw rate motion model. Since constant acceleration and constant yaw rate motion are assumed, we have a i = a i-n ω i =ω i-n .

[0099] Upsampling is used to synchronize low-frame-rate or delayed cloud information with high-frame-rate vehicle data, thereby improving the timeliness and accuracy of matching and fusion.

[0100] In some embodiments, the driving data of the cloud-sensed target and the driving data of the vehicle-mounted observed target are converted to the same spatial coordinate system to ensure consistency in subsequent matching and fusion.

[0101] It should be understood that the same spatial coordinate system can be the global coordinate system or the coordinate system of the autonomous driving device itself, i.e., the vehicle coordinate system.

[0102] It should also be understood that the driving data may include at least one of position, driving speed, and heading angle.

[0103] Figure 2 A schematic diagram illustrating the transformation relationship between the global coordinate system and the vehicle coordinate system provided in an embodiment of this application is shown. Figure 2 The embodiment provides a method for converting the driving data of the cloud-sensed target to the vehicle's coordinate system.

[0104] like Figure 2 As shown, in the global coordinate system ( X g O gY g Under these conditions, the lateral position, longitudinal position, heading angle, longitudinal velocity, and lateral velocity of the vehicle are respectively... x ego,g , y ego,g , θ ego,g , v egox,g , v egoy,g The target vehicles are respectively x obj,g , y obj,g , θ obj,g , v objx,g , v objy,g To convert the target vehicle's pose and velocity in the global coordinate system to the vehicle's coordinate system (… X loc O loc Y loc The corresponding quantity under ) x obj,loc , y obj,loc , θ obj,loc , v objx,loc , v objy,loc First, we need to calculate the position difference between the target vehicle and our own vehicle, using the following formula: (5) (6) Then, the position difference is transformed to the vehicle's coordinate system using a rotation matrix to obtain the target vehicle's position in the vehicle's coordinate system: (7) in, (8) The conversion between speed and heading angle can be expressed as: (9) (10) (11) The spatial transformation described above can eliminate the differences in position, velocity, and heading angle caused by different coordinate systems, providing a consistent reference basis for subsequent target matching.

[0105] Figure 3This illustration shows a flowchart of a vehicle-cloud data fusion method provided in an embodiment of this application. It should be understood that... Figure 3 The illustrated embodiment can be considered as Figure 1 A specific example of an embodiment. For example... Figure 3 As shown, the method includes the following steps.

[0106] S310: Preprocess the data sent from the cloud.

[0107] Specifically, the preprocessing operations include: S311, Anomaly Detection; S312, Upsampling; S313, Space Conversion.

[0108] S320: The preprocessed cloud-based data is correlated and matched with the vehicle-mounted observation targets.

[0109] S330, Determine if the match was successful.

[0110] If the match is successful, execute S331 to transmit the data from the cloud to the vehicle-mounted observation target and generate the predicted state sequence of the vehicle-mounted observation target; If the matching fails, S332 is executed, which outputs the unmatched cloud-sensing targets in the cloud-downloaded data separately to the downstream control system as a supplement to the sensing, and is particularly effective for situations such as occlusion, missed detection, and inaccurate identification at a distance.

[0111] Compared to existing methods, this application's solution integrates cloud-based data with perceived targets, supplementing perception, especially in cases of occlusion, missed detection, or inaccurate identification of distant objects. This allows for more flexible and efficient regulatory control strategies. Furthermore, it addresses latency and data anomalies that occur during the transmission of cloud-based data, ensuring effective integration of vehicle and cloud data and providing greater security for the use of vehicle and cloud data in regulatory control.

[0112] The foregoing mainly describes a vehicle-cloud data fusion method according to an embodiment of this application with reference to the accompanying drawings. It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially, these steps are not necessarily executed in the order shown in the figures. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of steps or stages in other steps. The following describes a vehicle-cloud data fusion device according to an embodiment of this application with reference to the accompanying drawings. For brevity, appropriate omissions will be made in the following description of the device; relevant content can be referred to in the relevant descriptions of the method above, and will not be repeated.

[0113] Corresponding to the method described in the above embodiments, Figure 4 The diagram shows a structural block diagram of a vehicle-cloud data fusion device 1000 provided in one embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0114] like Figure 4 As shown, the device 1000 may include: The receiving module 1001 is used to receive data sent from the cloud, which includes driving data of the cloud-sensed target.

[0115] The acquisition module 1002 is used to acquire driving data of the vehicle-side observed target, which is collected by sensors deployed on the autonomous driving equipment.

[0116] The matching module 1003 is used to match the cloud-sensing target with the vehicle-mounted observation target; The fusion module 1004 is used to fuse the driving data of the cloud-sensing target and the driving data of the vehicle-side observed target when the matching is successful, so as to obtain the predicted state sequence of the vehicle-side observed target.

[0117] In some embodiments, the device 1000 may further include a preprocessing module 1005 for preprocessing data sent from the cloud, such as performing steps S311-S313.

[0118] In some embodiments, the device 1000 may further include a virtual tracking module 1006 for performing S150.

[0119] In some embodiments, the matching module 1003 can also be used to perform S131-S134.

[0120] Figure 5 A schematic diagram of the hardware structure of a computer device provided in an embodiment of this application is shown.

[0121] The computer device may include a processor 7001 and a memory 7002 storing computer program instructions.

[0122] Specifically, the processor 7001 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0123] Memory 7002 may include mass storage for data or instructions. For example, and not limitingly, memory 7002 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 7002 may include removable or non-removable (or fixed) media, or memory 7002 may be non-volatile solid-state memory. Memory 7002 may be internal or external to the integrated gateway disaster recovery device.

[0124] In one instance, the memory 7002 may be a read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0125] The processor 7001 reads and executes computer program instructions stored in memory 7002 to achieve... Figure 1 The vehicle and cloud data fusion method in the illustrated embodiment.

[0126] In one example, the computer device may also include a communication interface 7003 and a bus 7004. Wherein, as... Figure 5 As shown, the processor 7001, memory 7002, and communication interface 7003 are connected through bus 7004 and complete communication with each other.

[0127] The communication interface 7003 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0128] Bus 7004 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 7004 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0129] Furthermore, in conjunction with the vehicle-cloud data fusion method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the vehicle-cloud data fusion methods in the above embodiments.

[0130] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the vehicle-cloud data fusion methods described in the above embodiments.

[0131] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0132] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0133] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0134] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A vehicle-to-cloud data fusion method applied to an automatic driving device, characterized in that, include: Receive data transmitted from the cloud, wherein the data transmitted from the cloud includes driving data of cloud-sensed targets; Acquire driving data of the vehicle-mounted observation target, wherein the driving data of the vehicle-mounted observation target is collected by sensors deployed on the autonomous driving equipment; Match the cloud-sensing target with the vehicle-mounted observation target; If a match is successful, the driving data of the cloud-sensing target and the driving data of the vehicle-mounted observation target are fused together to obtain the predicted state sequence of the vehicle-mounted observation target.

2. The method of claim 1, wherein, The matching of the cloud-sensing target with the vehicle-mounted observation target includes: The cloud-sensing target and the vehicle-mounted observation target are pre-matched to obtain multiple sets of candidate target pairs; For each pair of candidate targets, a matching cost is calculated based on the spatial position difference features and motion difference features between the cloud-sensing target and the vehicle-mounted observation target. The spatial position difference features include Mahalanobis distance and / or position difference, and the motion difference features include velocity difference and / or heading angle difference. A matching cost model is constructed based on the matching costs of the multiple sets of candidate target pairs; An optimized matching algorithm is used to solve the matching cost model globally to obtain the matching target pairs.

3. The method of claim 2, wherein, The matching cost model is a cost matrix. The optimization matching algorithm is used to globally solve the matching cost model to obtain matching target pairs, including: The cost matrix is ​​solved globally using the Hungarian matching algorithm. Candidate target pairs that meet the minimum global matching cost are selected from the multiple candidate target pairs and determined as the matching target pairs.

4. The method of claim 2, wherein, Before calculating the matching cost for each candidate target pair based on the spatial location difference features and motion difference features between the cloud-sensing target and the vehicle-mounted observation target, the method further includes: For each pair of candidate targets, the Mahalanobis distance and the position difference are calculated based on the observation state vector and covariance matrix of the vehicle-mounted observation target and the perception state vector of the cloud-based perception target. Candidate target pairs whose Mahalanobis distance exceeds a preset threshold and / or whose position difference exceeds a preset distance threshold are eliminated.

5. The method according to any one of claims 1 to 4, characterized in that, The process of fusing the driving data of the cloud-sensing target with the driving data of the vehicle-mounted observation target to obtain the predicted state sequence of the vehicle-mounted observation target includes: Based on the driving data of the vehicle-mounted observation target, the driving data of the cloud-sensing target are fused together, and the predicted state results of the vehicle-mounted observation target are updated using a filtering algorithm to generate the predicted state sequence of the vehicle-mounted observation target.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes: For cloud-sensing targets that fail to match, each failed cloud-sensing target is included in the tracking as a virtual target until the preset life threshold of the tracked cloud-sensing target is reached.

7. The method according to any one of claims 1 to 4, characterized in that, Before matching the cloud-sensing target with the vehicle-mounted observation target, the method further includes: Based on the received historical driving data of the cloud-sensing target, and with the sampling time of the driving data of the vehicle-side observed target as the target, the driving data of the cloud-sensing target is extrapolated and predicted using a motion model.

8. The method according to any one of claims 1 to 4, characterized in that, Before matching the cloud-sensing target with the vehicle-mounted observation target, the method further includes: The driving data of the cloud-sensing target and the driving data of the vehicle-mounted observation target are converted to the same spatial coordinate system. The driving data includes at least one of position, driving speed and heading angle.

9. The method according to any one of claims 1 to 4, characterized in that, Before matching the cloud-sensing target with the vehicle-mounted observation target, the method further includes: Anomaly detection is performed on the driving data of the cloud-sensing target, and driving data that meets preset anomaly conditions is filtered out. The preset anomaly conditions include at least one of the following: The driving speed exceeds the preset speed threshold; The sampling delay exceeds the preset delay threshold; Data was repeatedly sent beyond the preset frame count threshold. 10.A vehicle and cloud data fusion device configured in an autonomous driving apparatus, the device comprising: include: The receiving module is used to receive data sent from the cloud, which includes driving data of cloud-sensed targets; The acquisition module is used to acquire driving data of the vehicle-mounted observation target, which is collected by sensors deployed on the autonomous driving equipment. A matching module is used to match the cloud-sensing target with the vehicle-mounted observation target; The fusion module is used to fuse the driving data of the cloud-sensing target and the driving data of the vehicle-mounted observation target when the matching is successful, so as to obtain the predicted state sequence of the vehicle-mounted observation target.

Images